1 Introduction
There is a wealth of evidence that human judgment is often affected by motivational and cognitive factors (Reference KundaKunda, 1990; Reference Nisbett and RossNisbett & Ross, 1980). For example, people often rate themselves as being better than average (Reference Dunning, Meyerowitz and HolzbergDunning et al., 1989), and attribute failure to uncontrollable external factors while claiming credit for success (Reference Miller and RossMiller & Ross, 1975). In light of this, researchers have further sought to understand whether people are aware of their susceptibility to such biases, and demonstrated that self-assessments of one’s own bias are often biased as well.
Bias blind spot (BBS) is the phenomenon that people perceive themselves to be less susceptible to biases than others. Research on BBS builds on previous work on actor-observer attributions documenting people’s asymmetrical perceptions of themselves compared to others (Reference Jones, Nisbett and JonesJones & Nisbett, 1972; Reference Pronin, Gilovich and RossPronin et al., 2004), and it would seem that these misalignments extend to differing perceptions of one’s own biases compared to those of others.
BBS holds important practical implications. Doctors view themselves as being immune to the impact of gifts from pharmaceutical industry representatives on their own judgments, while perceiving their peers’ judgments to be strongly affected (Reference Dana and LoewensteinDana & Loewenstein, 2003). The illusion of objectivity associated with BBS has also been linked with discrimination in hiring (Reference Uhlmann and CohenUhlmann & Cohen, 2007).
If people are to improve their decision-making and overcome biases, they must first be able to recognize and understand their own biases (Reference Hansen, Gerbasi, Todorov, Kruse and ProninHansen et al., 2014). Yet, BBS suggests that recognizing one’s own biases is a tricky challenge, because people’s understanding of their own biases is influenced by the very same biases they aim to better understand, thereby resulting in an ironic “blind spot”.
We begin by introducing the literature on the BBS and the chosen article for replication, discussing the motivations for the current replication study. We then introduce free-will beliefs as a suggested extension and hypothesize about the relationship between free will beliefs and the bias blind spot effect.
1.1 Bias Blind Spot: Theoretical mechanisms
Bias blind spot has been theorized to be a result of the three possible sources: introspection illusion, naïve realism, and self-enhancement motives (Reference Pronin and KuglerPronin & Kugler, 2007).
The introspection illusion (Reference Nisbett and WilsonNisbett & Wilson, 1977) is people’s tendency to place more confidence in introspective information that they can access directly (Reference Ross, Ward, Brown, Reed and TurielRoss & Ward, 1996). People use introspection to access their own intentions, emotions, thoughts, and salient attitudes when seeking to understand and explain their own behaviors. Yet, when aiming to understand and explain others’ behaviors, others’ thoughts and feelings are not directly accessible and inferences are considered unreliable. People must resort to focusing merely on evaluating others based on their behaviors, which leads to differences in how people perceive their own behavior and decision-making process compared to how they perceive others’ (Reference Nisbett and RossNisbett & Ross, 1980). Many such biases in self-other perceptions have been documented over the years. For example, the widely discussed classic actor-observer bias (Reference Jones, Nisbett and JonesJones & Nisbett, 1972) is the perception that others’ behaviors reflect stable personality dispositions rather than contextual factors (Reference JonesJones, 1990).
Naïve realism (Reference Ross, Ward and ZannaRoss & Ward, 1995) describes people’s tendency to believe that their own perceptions of the world are more objective and accurate than others’. Because one’s own thoughts, attitudes, and beliefs come to mind readily and easily, they also appear to be devoid of subjective distortions. As a result, people can fairly easily detect subjectivity in others, yet fail to detect subjectivity in themselves (Reference Pronin and KuglerPronin, 2007; Reference Pronin, Gilovich and RossPronin et al., 2004; Reference Ross, Ward and ZannaRoss & Ward, 1995; Reference Wilson and DunnWilson & Dunn, 2004). Biases are often considered a deviation from rational objectivity, and it therefore follows that people would perceive themselves as being less biased than others.
Finally, people tend to construe reality in a way that would make for more positive self-evaluations. Biases are commonly perceived as sub-optimal, often referred to as being “irrational”, and so to maintain a positive self-image of being rational and objective, people would need to view themselves as being less biased (Reference Pronin and KuglerPronin & Kugler, 2007). Ironically, they also resist revising their views about being biased, and persist in their biased perceptions even after having been made aware of their biases (Reference Ehrlinger, Gilovich and RossEhrlinger et al., 2005; Reference Pronin and KuglerPronin, 2007).
Following the recent developments in psychological science and the growing recognition of the importance of reproducibility and replicability (e.g., Brandt et al., 2014; Open Science Collaboration, 2015; Van‘t Veer & Giner-Sorolla, 2016; Reference Zwaan, Etz, Lucas and DonnellanZwaan et al., 2018), we aimed to revisit the classic work on the BBS with independent replications. Very close direct pre-registered replications are needed to assess previous claims and obtain an accurate estimate of the effect size. We therefore embarked on a series of pre-registered close replications of Pronin et al.’s (2002) work, with added extensions to further our understanding of the phenomenon. Pronin et al. (2002) provided the first demonstration of the effect by examining people’s ratings of susceptibility to common biases in themselves and in others, and showed that people tend to perceive themselves as being less biased, even when being made aware of their potential bias.
We conducted very close replications and extensions of Pronin et al. (2002) to revisit their original findings. We reproduced their materials, addressed and improved their methods and analysis, and examined new directions. Replications contribute to theory specification and form the basis of further theory development (Reference Glöckner and BetschGlöckner & Betsch, 2011).
1.2 Choice of target article: Pronin et al. (2002)
We chose Pronin et al.’s (2002) studies for replication for their importance to the self-other asymmetry literature. Specifically, Pronin et al. (2002) found empirical evidence for the view that the sources of BBS are not restricted to self-serving motives. The findings emphasize the primary role of the introspection illusion and naïve realism on BBS effects. The article has had a significant impact on research in social psychology and judgment and decision-making. At the time of writing (March 2021), Google Scholar indicated 1129 citations of the article, laying ground to broader theories about human sense-making and objectivity (Reference Pronin and KuglerPronin, 2007; Reference Pronin, Gilovich and RossPronin et al., 2004). More recent theoretical assertions from Scopelliti et al. (2015) claim the BBS effect is a meta-bias that can be measured at the individual level. To the best of our knowledge, there are no published direct replications of Pronin et al. (2002).
Given the significance of Pronin et al.’s (2002) work, it is important to ascertain the replicability of the findings using high powered samples. Across the four studies reported in the original paper, the sample size ranged from 24 to 91, with an average sample size of 55 per study. Pronin et al. (2002) specifically argued that people’s introspection fails to detect unconscious processes involved in biases, whereas people’s introspection regarding own personal shortcomings — such as the tendency to procrastinate, the fear of public speaking, and the planning fallacy — are more visible and more accessible. Based on this argument, the authors predicted that the self-versus-others asymmetry is larger for biases than for personal shortcomings. Study 1 Survey 2 of Pronin et al. (2002) found empirical support for the view that people rate themselves, compared with others, to be less susceptible to biases, whereas there was no support for self-other asymmetry in ratings of personal shortcomings. Authors interpreted the findings as the demonstration of the stronger role of introspective illusions on BBS. The argument was that introspection may work to reduce self-other asymmetry in personal shortcomings, because shortcomings are more readily accessible and easier to contemplate than cognitive biases. However, it is possible that the original studies reported the lack of significant self-other differences in ratings on susceptibility to shortcomings because of the underpowered samples. Similarly, the same issues with power may have been the reason why they were able to detect the hypothesized effect in only four of the eight self-serving bias items tested. One of the goals of the current replication was to test the replicability of the BBS effects reported in Pronin et al. (2002) with high-powered samples.
In the current replication, we also respond to recent calls for the empirical testing of theories in non-Western populations. Scholars note that the psychology literature is lopsided, with 80% of study participants in psychology coming from Western, educated, industrialized, rich and democratic (WEIRD) societies (Reference Henrich, Heine and NorenzayanHenrich et al., 2010). Here, we also tested the replicability of the BBS effect among Hong Kong participants.
1.3 Extension: Free will beliefs and differences in self-other attributions
We extended our replication and literature on the BBS to examine associations with individual differences. So far, there is little work examining individual differences as predictors of BBS, although one example is by West et al. (2012) who reported failing to find support for associations between cognitive ability and sophistication and the BBS. We were specifically interested in possible associations with free will beliefs as laybeliefs regarding self-regulation and own choice and control (Reference Baumeister and MonroeBaumeister & Monroe, 2014).
Belief in free will is the generalized belief that human behavior is free from internal and external constraints across situations (Reference FeldmanFeldman, 2017; Reference Monroe, Dillon and MalleMonroe & Malle, 2014). Studies on the folk understanding of free will found that people normally associate free will with having choice, and understand free will as the absence of internal and external constraints (Reference BaumeisterBaumeister, 2008; Reference Feldman, Baumeister and WongFeldman et al., 2014; Reference Monroe, Dillon and MalleMonroe et al., 2014; Reference Monroe and MalleMonroe & Malle, 2010; Reference Vonasch, Baumeister and MeleVonasch et al., 2018). Free will beliefs have been found to be associated with a range of behavioral and psychological factors, such as academic and job performance (Reference Feldman, Chandrashekar and WongFeldman et al., 2016; Reference Stillman, Baumeister, Vohs, Lambert, Fincham and BrewerStillman et al., 2010), job satisfaction (Reference Feldman, Farh and WongFeldman et al., 2018), cooperation (Reference Protzko, Ouimette and SchoolerProtzko et al., 2016), and well-being and meaning in life (Reference Crescioni, Baumeister, Ainsworth, Ent and LambertCrescioni et al., 2016).
How would the concept of free-will relate to biases? First, by definition, people have limited to no control over their own biases, and these seem to mostly occur to regardless of one’s own choice, affecting individuals without their awareness. Therefore, stronger beliefs in one’s capacity to choose one’s path are likely to be associated with perceiving oneself as less biased.
Second, free will beliefs have been linked with counterfactual thinking (Reference Alquist, Ainsworth, Baumeister, Daly and StillmanAlquist et al., 2015; Reference Fillon, Lantian, Feldman and N’gbalaFillon et al., 2021), reflecting stronger introspection, which could contribute to learning to avoid similar mistakes in the future (Reference Feldman, Chandrashekar and WongFeldman et al., 2016). If that is the case, stronger self-reflection associated with stronger free will beliefs may indicate that the accentuated introspection illusion underlies the bias blind spot.
Third, in a previous study by Pronin and colleague about free will attributions, they demonstrated that “people believe they have more free-will than others” (Reference Pronin and KuglerPronin & Kugler, 2010). They explained their findings as a new take on the actor-observer bias, arguing that people perceive others’ behavior as more constrained (stable and unchanging personalities, hindering free will), whereas they tend to view their own behavior as intentional and reflective (responsive to situations, adaptive, and changing).
Fourth, free will beliefs were found to be associated with an overestimation of the influence of internal (dispositional and fixed) factors as compared to external (varying and adaptable) factors when evaluating others’ behavior (Reference Genschow, Rigoni and BrassGenschow et al., 2017).
Therefore, we explored the possibility that belief in free will would be positively related to the BBS.
We note that we did not pre-register a priori predictions regarding associations between free will beliefs and the bias blind spot regarding susceptibility to biases. Our predictions focused on self-other differences in personal shortcomings. Given the link between free will and differences in attributions when comparing oneself and others, stronger beliefs in one’s choice and in control over one’s own actions are likely to be associated with the perception that one’s own shortcomings are freely chosen, and less of a flaw. In comparison, stronger free will beliefs are also likely to be associated with the view that others’ shortcomings are a result of their dispositions, and therefore are more enduring flaws. Combined, these two possibilities are suggestive of a positive relationship between free will beliefs and self-other differences regarding personal shortcomings.
1.4 Chosen studies for replication: Studies 1 and 2
Our target article for replication, Pronin et al. (2002), consisted of three experiments, and of those we chose to focus on Studies 1 and 2.
In Study 1, participants separately rated their own and the average American’s susceptibility to eight biases and three personal shortcomings. Pronin et al. (2002) proposed and found support for three hypotheses: 1) participants rate themselves as less susceptible to biases than others, 2) there are no self-other differences detected for ratings of personal shortcomings (null hypothesis), and, therefore, 3) self-other differences are larger for biases than for shortcomings.
In Study 2, participants compared themselves to others on six personality dimensions, three positive and three negative. Findings were in support of the better than average effect, with participants rating themselves higher on the positive personality dimensions and lower on negative ones, in comparison to others. In support of the bias blind spot, a large majority of participants who exhibited the better than average bias insisted that their ratings were accurate or too modest, even after reading about the better than average effect and their potential bias.
2 Method
2.1 Pre-registrations and open data/code
In each of the replication studies, we first pre-registered the experiment on the Open Science Framework (OSF), and data collection was launched within one week of registration. Pre-registrations, disclosures, power analyses, and all materials are available in the Supplementary Materials. These, together with datasets and R/RMarkdown code, were made available on the OSF at https://osf.io/3df5s/. All measures, manipulations, and exclusions for this investigation are reported, and data collection was completed before analyses. Pre-registrations are all available on the OSF: Study 1a — https://osf.io/f4rb6/; Study 1b — https://osf.io/tez7m/; Study 2 — https://osf.io/4m35x/; Study 3 — https://osf.io/kqatx/.
2.2 Data collections
The present investigation included three data collections. We summarized an overview of the three studies in Table 1.
Table 1: Summary of studies
Studies 1a and 1b were meant as a pre-test of the effects in an undergraduate class covering topics of open science and social psychology, and were designed to replicate the bias blind spot (BBS) effect (Study 1 Survey 2) and the better than average effect (BAE) (Study 2), reported in the target study.
Study 2 was a separate replication of the BBS effect, designed by a different replication team, and with a larger, well-powered sample of different demographics. The data collection for Study 2 was conducted separately within two weeks of the data collection for Studies 1a/b.
In light of findings from the first two studies, a year later we followed up with a third data collection. Study 3 aimed to replicate and extend the findings, especially regarding the free will beliefs extensions proposed and tested in Study 2, using an even larger sample that would allow for detecting smaller effects. Also, in Study 3 we tested for replicability of both BBS and BAE reported in the target article.
We briefly introduce the studies below, and detailed procedures for each of the three studies are reported in the Supplementary Materials.
2.3 Study 1a and Study 1b
Undergraduate students taking a social psychology class at the University of Hong Kong designed the replication studies, and were invited to take part in their classmates’ studies as participants. A total of 75 students enrolled in the class were invited to participate, and of those 49 undergraduate students chose to take part in the online survey. Of those, we excluded four students who designed Study 1a and six other students who designed Study 1b, resulting in a sample of 45 for Study 1a (M age = 20.20, SD = 0.99; 31 females) and 43 for Study 1b (M age = 20.23, SD = 1.04; 32 females).
Participants in Study 1a were presented with descriptions of eight biases and three personal shortcomings in randomized order: self-serving attributions for success or failures, dissonance reduction after free choice, positive halo effect, biased assimilation of new information, reactive devaluation of proposal from one’s negotiation counterparts, perceptions of hostile media bias toward one’s group or cause, the fundamental attribution error, self-interest, procrastination, fear of public speaking, and the planning fallacy. Full descriptions are detailed in the Supplementary Materials. For each of the descriptions, participants rated their own susceptibility (biases: α = 0.58; personal shortcomings: α = 0.71) and perceived susceptibility of the average student at the university (biases: α = 0.73; personal shortcomings: α = 0.58). Participants responded to the following: “Indicate to what extent you believe that [you/others] show this effect or tendency?” on a nine-point scale (1 = not at all; 5 = somewhat; 9 = strongly).
Participants in Study 1b were presented with three positive and three negative personality traits in randomized order. The positive personality dimensions assessed were dependability, objectivity, and consideration ( α = 0.30). The negative personality dimensions assessed were snobbery, deceptiveness, and selfishness ( α = 0.60). The ratings were made on a 9-point scale (1 = much lower than the average student; 5 = same as the average student; 9 = much higher than the average student).
After rating their personalities, participants were briefed about the better than average effect and asked to reassess their personalities (1 - Objective measures would rate me lower on positive characteristics and higher on negative characteristics than I rated myself; 2 - Objective measures would rate me neither more positively nor more negatively than I rated myself; 3 - Objective measures would rate me higher on positive characteristics and lower on negative characteristics than I rated myself).
2.4 Study 2
A total of 310 American Amazon Mechanical Turk (MTurk) participants completed the study using TurkPrime.com (Reference Litman, Robinson and AbberbockLitman et al., 2017). Of those, seven participants were excluded based on pre-registered exclusion criteriaFootnote 1, leaving 303 participants included in the analysis (M age = 38.45, SD = 11.58; 166 females).
Participants first rated their free will beliefs on two scales (randomized order) and then rated their own and others’ susceptibility to biases and personal shortcomings. Display of conditions was counterbalanced in a 2 (reference: self vs. others) by 2 (attributes: biases and personal shortcomings) within-subject design (see the Supplementary Materials for more details and full measures). The materials used for testing BAE and BBS effects in Study 2 were the same as those of Study 1a.
For the extension, free will beliefs were measured before the replication using two free will belief subscales: Free Will Inventory (FWI) (Reference Nadelhoffer, Shepard, Nahmias, Sripada and RossNadelhoffer et al., 2014) (5 items; 1 = Strongly disagree, 7 = Strongly agree; α = 0.91) and Free Will and Determinism (FWD) personal agency subscale (Reference Rakos, Laurene, Skala and SlaneRakos et al., 2008) (4 items; 1 = Not true at all, 5 = Almost always true; α = 0.92). FWI was chosen as one of the most recently developed measurements of generalized free will beliefs, improving on many of the issues identified in earlier scales. FWD was chosen as it focuses specifically on free will beliefs in the self, which we thought might differ from the more generalized FWI scale. Full measures are provided in the Supplementary Materials.
2.5 Study 3
A total of 629 American MTurk participants completed the study using TurkPrime.com. Of those, eight participants were excluded based on pre-registered exclusion criteriaFootnote 2, leaving 621 participants included in the analysis (M age = 39.15, SD age = 11.88; 346 females). We prevented participants who took part in Study 2 from participating in Study 3.
The study included three parts: 1) free will beliefs, 2) replication of BBS effects (similar to Study 2), 3) replication of BAE (similar to Study 1b). Participants rated their own and other average Americans on susceptibility to biases (self: α = 0.79; others: α = 0.79) and personal shortcomings (self: α = 0.49; others: α = 0.55) (1 = not at all; 9 = strongly), and positive ( α = 0.45) and negative personalities ( α = 0.74) (1 = much lower than the average American; 5 = same as the average American; 9 = much higher than the average American).
In addition to the free will beliefs measures we collected in Study 2, we added an additional measure of generalized free will beliefs, using an older scale by Reference Paulhus and CareyPaulhus and Carey (2011). This was meant to help reexamine our interpretation of the extension findings in Study 2 about differences between free will beliefs in self, and generalized free will beliefs.
Free will beliefs scales included were: Free Will Inventory (FWI; 5 items; Reference Nadelhoffer, Shepard, Nahmias, Sripada and RossNadelhoffer et al., 2014) (1 = Strongly disagree, 7 = Strongly agree; α = 0.89), Free Will and Determinism personal will sub-scales (FWD; 8 items; Reference Rakos, Laurene, Skala and SlaneRakos et al., 2008) (0 = Not true at all, 4 = Almost a lways true; α = 0.74), and Free Will and Determinism plus scale (FAD+; 7 items; Reference Paulhus and CareyPaulhus & Carey, 2011) (1 = Not at all true, 5 = Always true; α = 0.85; recoded from a scale of 0 to 4 to match the original scale range).
2.6 Mini meta-analysis summary and evaluation criteria for replication design and findings
To summarize the replication results across the three samples, we conducted a mini meta-analysis of the effect sizes observed across the three replication studies (Reference Goh, Hall and RosenthalGoh et al., 2016; Reference Lakens and EtzLakens & Etz, 2017). The mini meta-analysis is not only helpful in succinctly summarizing the findings across studies but also in yielding a more precise estimate of the effect size (Reference Braver, Thoemmes and RosenthalBraver et al., 2014; Reference CummingCumming, 2014).
We characterize the current replications as “very close replications” based on the framework for classification of replications using the criteria by LeBel et al. (2018) (see Table S25 in the Supplementary Materials).
To interpret the replication results we followed the framework by LeBel et al. (2019). They suggested a replication evaluation using three factors: (a) whether a signal was detected (i.e., the confidence interval for the replication Effect size (ES) excludes zero), (b) consistency of the replication ES with the original study’s ES, and (c) precision of the replication’s ES estimate (see Figure S18 in the supplementary material).
3 Results
We first conducted replication tests mirroring the tests conducted by Pronin et al. (2002). Descriptive statistics are provided in Table 2. For the proposed extension, we calculated correlations between measures of free will beliefs and BBS measures. Detailed results for each of the three studies are reported in the Supplementary Materials section (see Table S3).
Table 2: Descriptive statistics for all studies
3.1 Susceptibility to biases and personal shortcomings
We conducted a series of one-tailed paired samples t-tests, and found that participants showed a general tendency to view themselves as less susceptible to biases across the three samples (Study 1a: M d = –0.75, t(44) = –4.54, p <.001, d z = –0.68, 95% CI [–1.01, –0.35]; Study 2: M d = –1.15; t(302) = –16.16, p < .001; d z = –0.93, 95% CI [–1.06, –0.79]; Study 3: M d = –1.80; t(620) = –32.04, p <.001, d z = –1.29, 95% CI [–1.39, –1.18]). We conclude these findings to be in support of the BBS hypothesis and the original findings.
Contrary to the predictions made in the original study, we found support for differences in ratings of personal shortcomings comparing self and others in Study 2 (M d = –0.52; t(302) = –5.22, p < .001; d z = –0.30 , 95% CI [–0.42, –0.18]) and Study 3 (M d = –0.73; t(620) = –10.54, p <.001, d z = –0.42, 95% CI [–0.51, -0.34]), yet with the weaker effect failing to show support for differences in Study 1a (M d = –0.29; t(44) = –1.13, p = .265; d z = –0.17, 95% CI [–0.47, 0.13]).
Finally, we found support for differences between the mean self-other bias asymmetry (across all biases) and the mean self-other personal shortcomings asymmetry (across all shortcomings) across all three samples (Study 1a: M d = –0.46, t(44) = –1.97, p = .055, d z = –0.29, 95% CI [–0.60, 0.01]; Study 2: M d = –0.62; t (302) = –6.39, p < .001, d z = –0.37, 95% CI [–0.48, –0.25]; Study 3: M d = –1.06; t (620) = –13.01, p <.001, d z = –0.52, 95% CI [–0.61, –0.44]).Footnote 3
3.2 Denying personal susceptibility to the better than average effect
We conducted one-sample one-tailed t-tests and found that participants rated themselves as having personality characteristics that are more positive (Study 1b: M = 5.74, t(42) = 5.09, p < .001, d z = 0.78, 95% CI [0.43, 1.11]; Study 3: M = 6.42; t(620) = 31.74, p < .001; d z = 1.27, 95% CI [1.17, 1.38]) and less negative (Study 1b: M = 4.16, t(42) = –4.55, p < .001, d z = –0.69, 95% CI [–1.02, –0.36]; Study 3: M = 3.21; t (620) = –30.38, p <.001; d z = –1.22, 95% CI [–1.32, –1.11]) than others’. We conclude support of the better than average effect.
We then proceeded to conduct chi-squared tests to examine denial of susceptibility to the better than average effect, comparing to a 50%-50% split. In Study 1b, we found that despite being made aware of their potential bias, only 9 of the 43 participants (21%) acknowledged any potential bias, leaving 79% of participants still claiming to be better than their average peers (χ 2 (1, N = 43) = 14.53, p < .001, d z = 1.43, 95% CI [0.69, 2.16]). We found similar results in Study 3, with only 109 of the 621 participants (18%) admitting bias, leaving 82% denying the bias (χ 2 (1, N = 621) = 261.53, p < .001; d z = 1.71, 95% CI [1.50, 1.91]).
We conclude that our findings are in support of denial of the better than average effect bias. The effect size of the replications (Study 1b: d z = 1.43; Study 3: d z = 1.27) was around twice the effect size of the original study (d z = 0.70).
Combining the findings of the three replications in a mini-meta analysis, we summarize the following: (1) participants rated themselves as less susceptible to biases than others, (2) participants rated themselves as less prone to personal shortcomings than others, (3) self-other differences for biases were larger than self-other differences for personal shortcomings, (4) participants rated themselves as possessing more positive personality characteristics and less negative personality characteristics than others, and (5) participants denied correcting for the better than average bias even after having been made aware of it.
3.3 Extension: Free will beliefs and shortcomings
We examined the link between free will beliefs and perceived personal shortcomings in self and others. We summarized our findings in Table 3.
Table 3: Summary of the extensions’ hypotheses (confirmatory hypotheses)
Note. r = Pearson correlation coefficient. BFW = belief in free will; FWI = Nadelhoffer et al. (2014); FAD+ = Reference Paulhus and CareyPaulhus and Carey (2011); FWD = Rakos, Laurene, Skala, and Slane, (2008). Hypotheses and related findings regarding BBS in susceptibility to bias are provided in the Supplementary Materials.
All free will beliefs scales were negatively correlated with perceived personal shortcomings in self (r ∼ –0.22 [–0.32, –0.11] to –0.09 [–0.17, –0.01]). However, for four out of five free will beliefs measures, we found no support for a link between free will beliefs and perceived personal shortcomings in others (r ∼ 0.00 [–0.11, 0.12] to 0.05 [–0.03, 0.13]; FWD in Study 3, r ∼ 0.11 [0.03, 0.18]). These findings seem to indicate that free will beliefs correlate more strongly with perceived shortcomings in self than in others. Supporting this notion, we examined perceived shortcomings in self and in others together, and free will beliefs negatively correlated with personal shortcomings self-other asymmetry (r ∼ –0.24 [–0.34, –0.13] to –0.16 [–0.24, –0.09]).Footnote 4
Additional exploratory findings are reported in the Supplementary Materials (See Table S24). These suggested that stronger perceptions of personal free will were associated with stronger bias blind spot, but associations were much weaker for generalized free will beliefs.
3.4 General summary: Mini meta-analysis of replication findings
We summarized the replications of Pronin et al. (2002) across the three samples using a mini meta-analysis in Table 4, and plotted the results in Figure 1. We ran the mini meta-analysis with a random-effects model method using the meta package in R (Schwarzer, 2007; see Figure S15-S16 for detailed forest plots in the Supplementary Materials section).
Table 4: Summary and comparison of mini meta-analysis effects of the replication to the original article
Note. Effect size is Cohen’s d. Replication summary directly based on LeBel et al., (2018) that considered the following three statistical aspects: (1) whether a signal was detected (i.e., whether the 95% confidence interval, or CI, represented here by the error bars, excludes 0); (2) the consistency of the replication effect-size (ES) estimate with that observed in the original study (i.e., whether the replication’s CI includes the original ES point estimate); and (3) the precision of the replication’s ES estimate (i.e., the width of its CI relative to the CI in the original study). NHST summary is based on the interpretation of a one-tail p-value with alpha set at 0.05. See Table S3-S4 (Figure S15-S16) of the Supplementary Materials for detailed results of three replication studies. BFW = belief in free will.
Figure 1: Comparison of findings in Pronin et al. (2002) to the replications’ mini meta-analysis summary.
We found support for self-other differences in perceived susceptibility to bias (d = –1.00 [–1.33, –0.67]), self-other differences in perceived personal shortcomings (d = –0.34 [–0.46, –0.23]), and the effects were stronger for perceived susceptibility to bias compared to perceived personal shortcomings (d = –0.43 [–0.56, –0.29]). We found support for better than average effects, with higher rating for positive personality dimensions (d = 1.05 [0.57, 1.54]), negative personality dimensions (d = –0.98 [–1.49, –0.47]), and denial of better than average effect (d = 1.69 [1.49, 1.88]).
Our evaluation of the replication findings based on the LeBel et al. (2019) criteria is summarized in the right column in Table 4 (separate analyses for each study are provided Table S4 of the Supplementary Materials). We conclude strong support for the main hypotheses of the target article, with effects comparable and at times larger than the original’s (see Figure 1).
4 Discussion
4.1 Replications of the bias blind spot
We conducted three close replication studies of the bias blind spot effect. Our findings were in support of the main predictions of the BBS effect. The findings of our replication samples are consistent in the samples from USA and Hong Kong. The current replication contributes to the theory development of BBS effects for three main reasons.
Firstly, we empirically qualify the theoretical assertions of Pronin et al. (2002). We addressed one of the potential methodological shortcomings, the underpowered samples in the original study. Using a larger sample of participants, we found support for self-other differences for personal shortcomings, whereas the original article hypothesized and argued findings were in support of a null effect. It is very likely that the null findings reported in the target article were mostly due to insufficient power. Specifically, in their Study 1, the authors expected weaker effects for self-other differences for personal shortcomings than for bias, yet failed to provide evidence for rejecting the null findings. We, on the other hand, did find support for self-other asymmetry in personal shortcomings. Pronin and colleagues (2002) interpreted the findings as being in support of the null, a demonstration that participants are as aware of their own personal shortcomings as they are of others’. Our replication findings suggest that there exists a bias also for personal shortcomings.
Pronin et al. (2002) failed to reject the null regarding specific hypotheses, and so their conclusion was that “in these cases, participants saw themselves as being just as flawed as their peers, if not more so” (p. 372), hinting about support for the null. We found support for rejecting the null regarding that specific case. Replicators often focus on findings that were able to reject the null hypothesis, which is understandable given that the vast majority of findings in published articles until 2011 were about rejecting the null. Yet, our findings here suggest that there might be value in attempting replications for null effects, in cases where an effect was found in one domain but not in another related domain, and especially so when the original sample size was severely underpowered and there were reasons to believe there should have been an effect.
Despite the above divergence in findings, we believe that the theoretical grounds of the target paper remain relevant and that the replications are in support of the original theory and findings, as the replication results lend support for the main introspection hypothesis that self-other differences are larger for susceptibility to bias than for personal shortcoming. Our main recommendation for those studying the BBS is to revise the original conclusions for a null effect for personal shortcomings into expecting an effect, though a weaker effect than for biases.
Secondly, we contribute to understanding of the BBS effect in a cross-cultural setting. We find support for BBS effects across samples from the USA and Hong Kong. For instance, Study 1 used a sample of non-naive students from Hong Kong, who were familiar with heuristics and biases in judgment and decision-making. The sample was slightly underpowered, and yet we were able to detect the effects reported in the classic article. Despite knowledge of other biases, and despite possible cultural differences, the findings were highly consistent with the findings in the classic article and the two MTurk samples. We advise caution in over-interpreting these, given the small sample and the singular bias under examination, yet it offers some support for BBS being robust enough to withstand concerns such as non-naivete and cross-cultural differences. Concerns around cultural relativity of the findings in psychology seems to be less of a concern with BBS effects (Reference GergenGergen, 1973; Reference Henrich, Heine and NorenzayanHenrich et al., 2010).
Thirdly, going beyond the BBS, we believe that our investigation offers additional insights into the BBS effect. In Studies 2 and 3, we examined the link between free will beliefs and the bias blind spot effects (see summary in Table 3). Overall, our findings supported our predictions that belief in free will would be associated with stronger self-other differences in perceived personal shortcomings (r ∼ 0.16–0.24). The differences between self and other ratings of personal shortcomings were mostly driven by the negative association between free will beliefs and personal shortcomings in self. We also found support for an association between free will beliefs and the BBS in susceptibility to bias (r ∼ 0.17–0.22), but only with for the free will beliefs scale that concerned the self, rather than in general. In Study 2, these findings were discovered in exploratory analyses, and in Study 3 we pre-registered the predictions and found very consistent effects, suggesting that these were not accidental.
The results suggest that free will belief associations are more about viewing oneself to be free of biases and shortcomings, and less about viewing others as more susceptible to biases and shortcomings. This indicates that free will beliefs in reference to the self are related to the nature of introspection accessed, complementing previous work that argued that free will beliefs impact fundamental socio-cognitive processes (Reference Genschow, Rigoni and BrassGenschow et al., 2017). Our take-aways from the extensions are the following: 1) free will beliefs are associated with self-other differences in perceived personal shortcoming, 2) free will beliefs in reference to the self are associated with BBS regarding susceptibility to bias, and 3) different free will beliefs scales gauge different aspects of agency. We advise free will beliefs researchers to incorporate both types of scales and make predictions differentiating between personal and generalized free will beliefs. It is possible that these differences would be especially relevant for biases regarding differences in attributions or perceptions comparing self to others. Overall, given the findings, we see much promise in the direction of examining links between free will beliefs and attribution biases.
It is important to note that our evidence for the relation between belief in free will and BBS effects is correlational only. Therefore, current findings do not allow for causal inferences. However, future research may aim to reproduce our findings by implementing experimental designs that manipulate free will beliefs. Furthermore, future research could extend current work in various ways. For example, future work can explore the role of belief in free will on the tendency to attribute responsibility or blame to people susceptible to cognitive biases.
Open access
