Working memory capacity and the risky-choice framing effect: A preregistered replication and extension of Corbin et al. (2010)

Abstract

While working memory capacity is associated with superior performance on a number of tasks, could it paradoxically sometimes be associated with suboptimal performance? Corbin et al. (2010, Judgment and Decision Making 5(2), 110–115) found that, in a between-subjects design, higher WMC is associated with a larger risky-choice framing effect, traditionally conceived of as a departure from rational principles. Such surprising findings are of potentially great theoretical importance; yet the original study was underpowered. In this registered report, we aimed to replicate and extend the original findings, by conducting an online experiment among 425 North Americans. To extend the findings beyond the specific single tasks used in the original study, we used three WMC tasks with different processing components and six framing problems involving human lives. In a close replication, the frame significantly interacted with neither the Ospan short absolute score nor the Ospan short partial score in predicting ratings on the disease-framing problem. Similarly, in an extended replication, a composite WMC score did not significantly interact with the frame in predicting ratings on three framing problems involving human lives. The Bayes factors showed that the data were 3 to 10 times more likely under the null hypothesis of no interaction between WMC and frame. Taken together, these findings show an absence of association between the between-subjects risky-choice framing effect and WMC. This outcome is compatible with four out of the six theoretical accounts we considered, and is uniquely predicted by the default-interventionist dual-process account and the pragmatic inference account. Further research can more rigorously pit conflicting predictions of these accounts against each other.

1. Introduction

When people deal with risk, their choices are sometimes influenced by apparently irrelevant factors, such as the way the choice options are presented. In particular, people tend to choose a safe option when the information is presented as potential gains, but they prefer a risky option when the same information is presented as potential losses—a phenomenon known as the risky-choice framing effect (Levin et al., 1998; Tversky & Kahneman, 1981, see the Appendix for examples). The effect is well-established (Klein et al., 2014; Kühberger, 1998) and is typically treated as a deviation from rational principles (e.g., Tversky & Kahneman, 1986; but see Mandel, 2014). Although considerable research has been conducted on the individual differences in the effect (e.g., Lauriola & Levin, 2001; Levin et al., 2002; Simon et al., 2004), it has sometimes reported contradictory results (e.g., regarding the effect of cognitive style, Smith & Levin, 1996 vs. LeBoeuf & Shafir, 2003 and Levin et al., 2002; or regarding gender effects, Fagley & Miller, 1997 vs. Reyna et al., 2011). Other times, the available evidence is based on single studies using relatively small samples. For instance, Corbin et al. (2010) reported a larger framing effect among participants with higher versus lower working memory capacity (WMC), as measured by the automated operation span task (Ospan, Unsworth et al., 2005). This finding is surprising because most relevant research either found that cognitive capacity is positively associated with adhering to rational principles or failed to find any association. The finding is also inconsistent with a dominant account of the framing effect, namely the default-interventionist dual-process account (Kahneman, 2000, 2003), though it could be accounted for by alternative accounts such as the fuzzy-trace theory (Corbin et al., 2015; Reyna, 2012). We replicated and extended Corbin et al.’s (2010) study to provide more robust evidence, and thus better inform theoretical accounts about the relationship between WMC and risky-choice framing.

1.1. Empirical evidence

Working memory refers to the set of processes that hold a limited amount of information in a highly available state to be used in thoughts and actions (Cowan, 2017; Oberauer et al., 2018). Working memory plays a central role in cognitive processing, in particular in deliberate cognition (Oberauer et al., 2018). Although there are many definitions of working memory, and relatedly various experimental paradigms to study it (Cowan, 2017), some individuals perform consistently better than others on various working memory tasks. These individuals are said to have higher WMC. Performance on WMC measures is also strongly associated with measures of general intelligence (Oberauer et al., 2018), to the point that researchers of individual differences in judgment and decision-making often put them into the same category of ‘cognitive ability/capacity’ (e.g., Corbin et al., 2015; Evans & Stanovich, 2013; Stanovich & West, 2000).

Higher cognitive capacity is associated with better performance on a host of tasks related to probability judgment and decision-making under risk (Brevers et al., 2012; Cokely & Kelley, 2009; Corbin et al., 2015; Del Missier et al., 2013; Dougherty & Hunter, 2003; Starcke et al., 2011). Recently, Burgoyne et al. (2023) found that a latent rationality factor, composed of base rate, conjunction fallacy, and Wason selection tasks, was strongly associated with intelligence (r = 0.56) and WMC (r = 0.44). Yet, some judgment and decision-making tasks failed to display associations with cognitive capacity (see Stanovich & West, 2008, p. 686, for an illustrative summary of both positive and null findings). In a related series of studies, McElroy et al. (2020) found that more thought was associated with improved decision-making for complex tasks but not for simple ones, including risky-choice tasks.

Similarly, research on the risky-choice framing effect in particular has either found a negative association with cognitive capacity or failed to find any association. Whenever a negative association was found, it was always in within-subjects designs, where participants see both versions of the same problem at different points in time (Bruine de Bruin et al., 2007). Even when using within-subjects designs, research has sometimes reported mixed findings (Del Missier et al., 2012; Stanovich & West, 1998) or failed to find a significant association altogether (Toplak et al., 2014). Research varying the frame between subjects (as originally done by Tversky & Kahneman, 1981; and also by Corbin et al., 2010) has failed to find reliable differences in framing effects related to cognitive capacity (Stanovich & West, 2008). The framing effect was also not larger under memory load compared with no load (Whitney et al., 2008), suggesting that working memory considerations play little role in framing tasks at least in some contexts.

In contrast to negative or null findings, results like Corbin et al.’s (2010), pointing to a positive association between cognitive capacity and the framing effect, are rare. Stanovich and West (2008) found a descriptively larger framing effect among individuals with higher scores on the Scholastic Assessment Test (SAT), as an index of cognitive ability. However, this finding was not statistically significant and was reported as a failure to find an association. Using a monetary task, Urs et al. (2019) failed to find a main effect of frame, and yet found that high-WMC individuals were more risk-seeking than low-WMC individuals under the loss frame when there was a large value at stake. The authors interpreted this finding as indicating a larger bias among high-WMC individuals, partially in accordance with Corbin et al. (2010). However, it is also possible that high-WMC individuals made superior choices because the risky options were also of a higher expected value than the sure options.

The strongest evidence supporting Corbin et al.’s (2010) findings comes from a further study by Corbin (2013, Experiment 1) that successfully replicated, using a larger sample (N = 161), the larger framing effect among higher-WMC individuals. However, the effect was present only among individuals scoring low in numeracy. Furthermore, the effect was not significant when using a score combining participants’ choice (Program A vs. B) and their confidence. Under high memory demands, higher-WMC individuals also displayed a smaller rather than larger framing effect compared with lower-WMC individuals (Corbin, 2013, Experiment 2). Rather than unanimously confirming Corbin et al.’s (2010) findings, Corbin’s (2013) findings present a mixed picture of possible moderators affecting the relationship between framing and WMC in either direction.

In sum, Corbin et al.’s (2010) findings of a positive association between cognitive capacity and the framing effect are corroborated by few studies, whereas most research has either shown a negative association or failed to find any association. Yet, if a positive relationship exists, the theoretical implications for advancing our understanding of the framing effect are of great importance.

1.2. Competing theoretical predictions

1.2.1. Dual-process theories

The framing effect was originally described and accounted for in the context of prospect theory (Tversky & Kahneman, 1981, 1986). Prospect theory proposes that people tend to evaluate the options relative to a neutral reference point, which shifts depending on how the options are presented. In the widely used disease problem (Tversky & Kahneman, 1981, see the Appendix), reading about lives to be saved induces a reference point of zero lives saved, so the options are evaluated as potential gains. By contrast, reading about people who would die induces a reference point of zero people dying, so the options are evaluated as potential losses. In addition, following a basic psychophysical law, people are decreasingly sensitive to changes of magnitude as the magnitude itself increases, which leads to risk aversion in the domain of gains and risk seeking in the domain of losses (Tversky & Kahneman, 1981). Dependence on reference points and diminished sensitivity jointly produce the risky-choice framing effect.

The prospect theory’s interpretation of the framing effect was incorporated into the dual-process framework of thinking (Kahneman, 2011; Stanovich & West, 2000), distinguishing between an automatic and effortless System 1/Type 1 thinking and a conscious and effortful System 2/Type 2 thinking that is, crucially, dependent on the limited cognitive capacity (Evans & Stanovich, 2013). In particular, Kahneman (2011) regarded both reference point and diminishing sensitivity as features of System 1. Kahneman’s (2003, 2011) account of the framing effect adheres to the so-called default-interventionist subtype of dual-process theories (Evans & Stanovich, 2013), which propose that an initial, default response to thinking problems is always provided by System 1. System 2 then intervenes, but only if it detects a conflict between the System 1 intuition and some normative consideration (Kahneman, 2000). Consequently, individuals who are better able and willing to think would do better on thinking tasks only if they are provided with a cue pointing to such conflict. When framing problems are presented between subjects, no cue is provided to realize that an alternative framing of the same problem exists (Kahneman, 2000, 2003). Both cognitively sophisticated and less sophisticated individuals would thus be equally likely to fall prey to the effect. By contrast, within-subjects designs help establish the equivalence between the two frames, and thus, provide an advantage to individuals of higher cognitive capacity, who are more willing or able to recognize the equivalence, recall their initial response, and be consistent in their responses. Thus, depending on design, the default-interventionist view accounts for the different patterns of associations between cognitive capacity and the framing effect (see also Stanovich & West, 2008).

Other dual-process accounts might also be relevant to the association between the framing effect and cognitive capacity. For instance, the parallel-competitive approach (Sloman, 1996) proposes that Type 1 and Type 2 processes operate in parallel and provide competing responses to a problem. If Type 2 processes are involved from the start, it is possible that at least some participants spontaneously consider different frames, resulting in a diminished framing effect. Since Type 2 processes depend on cognitive capacity, those of higher capacity should be more likely to resist framing even in between-subjects contexts. A parallel-competitive account thus predicts a negative association between cognitive capacity and the framing effect; it is largely incompatible with an absence of association. These predictions are supported by evidence that asking for a rationale, thus presumably engaging Type 2 thinking, decreases the effect even in between-subjects contexts (McElroy & Seta, 2003; Miller & Fagley, 1991; Takemura, 1994, Experiment 1), while presenting the task under time constraints, thereby preventing Type 2 thinking, increases it (Guo et al., 2017; Takemura, 1994, Experiment 2; but see Igou & Bless, 2007, for findings in the opposite direction).

A more recent subtype of the dual-process framework is the hybrid view (De Neys & Pennycook, 2019), which posits that adherence to rational norms is often due to Type 1 ‘logical’ intuitions rather than deliberate Type 2 thinking. Yet, these logical intuitions result from the automatization of procedures that were originally deliberate (De Neys & Pennycook, 2019), ensuring better intuitions for those of higher cognitive capacity (Thompson et al., 2018). Therefore, the best prediction of the hybrid view would be a negative association between WMC and the framing effect. Unlike rules of logic, however, it is not clear what specific procedures have to be automatized to prevent the between-subjects framing effect. This might be one reason why framing problems are notably absent from empirical tests of the hybrid account, a reason that also makes the account compatible with an absence of association.

1.2.2. Fuzzy-trace theory

A different account for the framing effect is proposed by fuzzy-trace theory (Corbin et al., 2015; Reyna, 2012; Reyna & Brainerd, 1991). Fuzzy-trace theory proposes that people represent information in both verbatim detailed form and gist form, keeping only the essential meaning. In the case of the between-subjects framing effect, verbatim information is typically useless because it leads to the same numbers (e.g., 200 = 1/3 × 600). Therefore, people have to rely on gist information. Although different levels of gist representations exist, fuzzy-trace theory predicts that people would rely on the simplest representation sufficient to elicit a preference (Corbin et al., 2015; Reyna, 2012). This lowest-level gist representation is provided by translating the numerical information into the fuzzy categories of ‘some’ and ‘none’. In particular, people would prefer ‘saving some’ (Program A) to ‘saving some vs. saving none’ (Program B) in the gain frame. However, they would prefer ‘some dying vs. none dying’ to ‘some dying’ in the loss frame.

This natural tendency of using gist representation can be overridden in within-subject contexts, allowing individuals to recognize the equivalence between the different versions of the same problem. The more able individuals are to inhibit the gist-based response, the more likely they will be to exhibit consistent responses (Corbin et al., 2015). Thus, similar to the dual-process accounts and consistent with empirical findings, fuzzy-trace theory predicts that individuals of higher cognitive capacity would resist within-subjects framing better. However, in a between-subjects context, the only helpful interpretation is provided by the gist. Therefore, fuzzy-trace theory predicts no relationship between cognitive capacity and between-subjects framing (Corbin et al., 2015). Fuzzy-trace theory also interprets the between-subjects framing effect as ‘an indicator of advanced processing’ (Corbin et al., 2015, p. 86), and is thus compatible with Corbin et al.’s (2010) findings for a higher framing effect among the more cognitively able (Corbin et al., 2015; see also Reyna et al., 2014). Indeed, recent independent research has provided evidence that higher WMC is positively associated with both gist and verbatim encoding (Nieznański & Obidziński, 2019).

1.2.3. Pragmatic accounts

While dual-process accounts and fuzzy-trace theories diverge in many respects, they agree that the framing effect violates normative principles (Chick et al., 2016; Kahneman, 2003). There are alternative approaches that, however, question this interpretation. For instance, Mandel (2014) argued that people presented with the framing tasks interpret some quantifiers as lower bounds. That is, when reading the gain-framed Program A of the disease problem, people infer that at least 200 people will be saved. Under this interpretation, Program A is superior to Program B. Similarly, having at least 400 people dying is inferior to Program B. Since this account treats the framing effect as resulting from pragmatic linguistic inferences, it posits that cognitive sophistication is not related to the size of the effect (Mandel & Kapler, 2018).

1.2.4. The intensified context shift hypothesis

Unexpected theoretical support for Corbin et al.’s (2010) findings comes from research on directed forgetting (Delaney & Sahakyan, 2007) which found that high-WMC participants recalled fewer items from a word list than low-WMC participants after an instruction to change the context (i.e., to imagine walking through one’s childhood home). Delaney and Sahakyan accounted for the findings with the intensified context shift hypothesis, whereby higher-WMC individuals are better able to access the context in which information was encoded. This advantage, however, comes at the cost of being more dependent on the context during retrieval. Over-reliance on irrelevant context information has also been proposed to be a major mechanism underlying the framing effect (Stanovich et al., 2016). To the extent that encoding and retrieval are also involved in solving a framing problem, high-WMC individuals might be more affected by the contextual information within each frame and thus show a larger framing effect, a prediction also noted by Corbin et al. (2010).

Table 1 summarizes the predictions of the competing accounts. Two accounts are compatible with more than one outcome, but no outcome is compatible with all accounts. Providing a well-powered response to the association question can inform alternative accounts and suggest revising those that are not supported by evidence.

Table 1 Competing predictions about the association between cognitive capacity and the between-subjects framing effect

Note. Best prediction in bold.

1.3. The present study

To closely replicate Corbin et al.’s (2010) experiment, we used the disease problem (Tversky & Kahneman, 1981) and a version of Ospan (Oswald et al., 2015; Unsworth et al., 2005), which allowed us to directly replicate the original statistically significant frame by Ospan score interaction effect on the response to the disease problem. Although Corbin et al. used the absolute Ospan score, researchers involved in devising the automated complex span tasks recommended the partial score because it has higher internal consistency, correlates better with measures of fluid intelligence, and makes more sense in terms of test theory (Redick et al., 2012).¹ Therefore, we predicted a significant frame by WMC interaction both when using the absolute Ospan score (Hypothesis 1a) and the partial Ospan score (Hypothesis 1b).

To further test the validity and generalizability of the findings, we added the following extensions to the original method. First, to test whether the effect would generalize beyond the disease problem, we included five more risky-choice framing problems (Berthet, 2021; Fagley et al., 2010; Simon et al., 2004). Similarly, we tested if the effect generalizes beyond Ospan. Ospan contains variance from both WMC and the task itself, meaning it also measures factors unrelated to WMC, such as the speed at solving math problems (Foster et al., 2015). In addition, Ospan has been generally found to have smaller loadings on a latent WMC factor compared with other complex span tasks (Draheim et al., 2018). Accordingly, we administered shortened versions of the automatic reading span (Rspan), and automatic symmetry span (SymSpan) and combined their scores into a single WMC score which is more valid than using Ospan alone (Oswald et al., 2015). For our extended replication, we tested if ratings on the framing problems would be significantly predicted by a frame by WMC interaction when a composite WMC task is used (Hypothesis 2).

The Stage 1 registered report, including introduction, method, and planned analyses, was preregistered at the Open Science Framework at November 7, 2002, https://osf.io/grp5m, and updated on April 28, 2023, https://osf.io/hw2sm.²

2. Method

2.1. Sampling plan

For the close replication, we performed power simulations by adapting guidelines originally provided for mixed-effects models (DeBruine & Barr, 2021) to the simpler general linear model used in the original study (Corbin et al., 2010). One set of simulations, using parameters extracted from the original data, showed that the original study had a 59% statistical power to detect a significant frame by Ospan interaction.³ This estimate converged with the estimate of 56% power from analytical power analysis using the MBESS R package (Kelley, 2021).

In another set of simulations, we determined our smallest effect of interest (SESOI) by estimating the effect size that the original study had a 33% power to detect (Simonsohn, 2015). To incorporate predictions from various theoretical accounts, we also considered negative interactions. Results showed that the original study had 33% power to detect betas of the interaction term of sizes −0.033 and +0.034. We then performed a third set of simulations to estimate the sample size needed to detect SESOI with 80% power. To this aim, we varied the number of participants from 200 to 400, in steps of 50, and beta estimates for the interaction term from −0.05 to 0.05, in steps of 0.02 (5,000 simulations per combination). A study with 400 participants would have more than 80% power to detect an interaction equal to ±0.03, which is roughly equal to SESOI (Figure 1).

Figure 1 Close replication: Power to detect the interaction effect at varying sample sizes and values of the interaction term (beta) (5,000 simulations per combination).

For the extended replication, we considered using several framing problems. Accordingly, in our simulations, we used mixed-effect modeling that allows using all individual responses in an unaggregated form rather than averaging by participant (Brown, 2021). Mixed-effects modeling is a more flexible analytical approach than linear regression and is related to higher statistical power and lower Type I error rate (Baayen et al., 2008; Barr et al., 2013). In our power simulations (DeBruine & Barr, 2021), we varied the number of participants from 200 to 400, in steps of 50, the number of framing tasks from three to six, and the beta of the interaction term, in steps of 0.02 (1,000 simulations per combination). The simulations showed that a sample size of 400 participants would ensure 90% power or more to detect SESOI (Figure 2). This number was consistent with the estimates for the close replication. We also performed a Bayes Factor Design Analysis for fixed-N designs (Stefan et al., 2019) using the Shiny app the authors provided for the purpose (http://shinyapps.org/apps/BFDA/). Results showed that 200 participants per group would provide at least 85% probability to detect a Bayes factor (BF) of at least 3 (in case of true effect) or 1/3 (in case of null effect) using either informed or default priors and an expected standardized effect size of 0.35.⁴ We thus decided to set 400 as our target sample size and six as the number of framing problems. All files to reproduce simulations are available at https://osf.io/ektfd/.

Figure 2 Extended replication: Power to detect the interaction effect at varying numbers of participants, numbers of framing items, and beta values of the interaction term (1,000 simulations per combination).

2.2. Participants

Participants were 562 North Americans recruited via Prolific. We used two Prolific workspaces, one funded by Sofia University and the other funded by Florida Gulf Coast University. Participants were compensated by Euro 7.50 at the former workspace and by USD 9.60 at the latter. Five hundred and fifty participants provided full data. One hundred and twenty-five participants were excluded who

• • indicated a nationality other than Canada/USA (n = 0);

• • indicated native language other than English (n = 0);

• • completed the experiment multiple times, as indicated by identical IP addresses and demographic information (n = 1);

• • responded ‘yes’ to the question asking about the familiarity with the framing tasks (n = 111);

• • were less than 95% likely to be above the guessing level on the processing component of the three WMC tasks overall, that is, with overall accuracy below 59.5% (Richmond et al., 2021, and R code provided by their anonymous reviewer) (n = 13).

The final sample consisted of 425 participants (207 female, 214 male, 4 other or prefer not to disclose), mean age 42.56 years (SD = 13.75, Min = 20, Max = 78, Med = 40). Participants were randomly assigned to a gain (n = 222) or loss (n = 203) condition.

2.3. Materials

2.3.1. WMC measure

The shortened versions of Ospan, Rspan, and Symspan (Oswald et al., 2015) were used, as described below. Each of these complex span tasks consists of both a storage component (i.e., items to be recalled) and a processing component (i.e., tasks involving math operations, sentence comprehension, or symmetry judgments). As practice trials for each task, participants went through the storage component alone, the processing component alone, and then the processing component followed by the storage component. For the purpose of the study, the tasks were scripted in PsyToolkit (Stoet, 2010, 2017).

Operation span. Participants were given a set of simple arithmetic operations. For each of them, they had to indicate whether a suggested answer was true or false and were then presented a letter to remember. At the end of each trial, participants were shown a 4 × 3 letter matrix and asked to click on the letters in the same order as the one they were presented in. Set sizes ranged from 4 to 6, with each set size being administered three times (45 processing-storage trials in total).

Reading span. Participants were asked whether various sentences (of length 10–16 words in English) made sense, with each of them followed by a letter to be remembered and consequently recalled at the end of the set. Set sizes ranged from 4 to 6, with each set size being administered three times (45 trials in total).

Symmetry span. Participants were presented with an 8 ×8 matrix of white and black squares and had to indicate whether the pattern was symmetrical along the vertical axis. After each matrix, the participants were shown a single red square positioned in a 4 × 4 matrix to be remembered and consequently recalled at the end of the set. Set sizes ranged from 3 to 5, with each set size being administered three times (36 trials in total).

Scoring. The complex span tasks are typically scored in two ways. The absolute score is the sum of correctly recalled items in perfectly recalled sets; the partial score is the sum of all correctly recalled items (Unsworth et al., 2005). In line with the original study (Corbin et al., 2010), we used the Ospan absolute score for the close replication (Hypothesis 1a). Yet, as partial scores typically have superior psychometric characteristics than absolute scores (Redick et al., 2012), we also used partial scores as a robustness check in the close replication (Hypothesis 1b) and as the only score in the extended replication (Hypothesis 2).

2.3.2. Framing tasks

Six framing problems were selected after Fagley et al. (2010) and Berthet (2021), namely the disease (adapted from Tversky & Kahneman, 1981), civil defense (adapted from Fagley & Miller, 1997; see also Fischhoff, 1983), cancer treatment (adapted from Fagley & Miller, 1987), traffic accident (adapted from Wang, 1996), African village, and derailed train (adapted from Svenson & Benson III, 1993). These problems have shown good internal consistency in previous research (Berthet, 2021; Fagley et al., 2010). Each problem presented a choice between a sure and a risky option of equal expected values. An example is the disease problem:

Imagine that the USA/Canada is preparing for the outbreak of an unusual disease, which is expected to kill 600 people. Two alternative programs to combat the disease have been proposed. Assume that the exact scientific estimates of the consequences of the programs are as follows:

(Gain frame)

If program A is adopted, 200 people will be saved.

If program B is adopted, there is a 1/3 probability that 600 people will be saved and a 2/3 probability that no people will be saved.

(Loss frame)

If program A is adopted, 400 people will die.

If program B is adopted, there is a 1/3 probability that nobody will die and a 2/3 probability that 600 people will die.

Which of the two programs would you favor?

All the problems were related to human lives to keep constant the potentially important factor of task content in risky-choice framing problems (Fagley & Miller, 1997). Following Fagley et al.’s (2010) recommendations, the number of lives at risk was constrained to be on roughly the same scale (varying from 100 to 600). The chances for no one saved in the probabilistic option were 2/3 in half the problems and 3/4 in the other half. In line with Corbin et al. (2010), participants indicated their preference on each task on a 7-point scale ranging from 1 (definitely would favor A) to 7 (definitely would favor B). Option A was always the safe option and Option B was always the risky option. The problems are listed in the Appendix.

2.3.3. Familiarity with the framing tasks

Since familiarity with the framing tasks reduces the framing effect (Rachev et al., 2021), participants were asked: ‘Have you seen any of these choice problems before?’, with response options ‘yes’ and ‘no’.

2.4. Procedure

Data were collected online using PsyToolkit (Stoet, 2010, 2017). Participants provided informed consent and filled in demographic information (nationality, gender, age, educational status, and major). They then completed the complex span tasks and the framing problems, presented in separate blocks. Unlike Corbin et al. (2010) who used a fixed order with Ospan always coming first, we counterbalanced the order of the WMC and framing blocks. This was done to prevent artifacts of task order (e.g., individuals with lower WMC might have been more exhausted by Ospan and consequently paid less attention to the framing problem, resulting in a smaller framing effect among these participants). To ensure close replication, Ospan and the disease problem were always kept first in their blocks, thus preventing carryover effects.

Within the WMC block, Ospan was followed by Rspan and then by Symspan. Within the Framing block, the five tasks following the disease problem were presented in a random order. We also exploratively checked if the problems were successful in eliciting a framing effect within subjects, as opposed to evoking a risk-seeking or risk-averse attitude regardless of the frame. To this aim, after rating the six problems in their assigned frame, participants were asked to rate two of the problems in the alternative frame.⁵ Participants were explicitly told that the aim was to check if it mattered to them whether the problems are stated as gains or losses. Immediately after the last framing problem came the familiarity question. The median completion time was 49 min.

Table 2 summarizes the differences between Corbin et al.’s (2010) method and our method.

Table 2 Differences between Corbin et al.’s method and our method

^aThe full-length Ospan task (Unsworth et al., 2005), as used by Corbin et al. (2010), administers three trials with set sizes ranging from 3 to 7, while the shortened version includes two trials with set sizes ranging from 4 to 6.

2.5. Piloting

To test if the script worked as intended, we ran a pilot study on (N = 30) participants. It also served as a pre-test for the framing and WMC measures. No issue has arisen during the piloting. The pilot dataset and summary statistics are available at OSF, https://osf.io/ektfd/.

3. Results

3.1. Summary statistics

Means, standard deviations, and intercorrelations are displayed for the framing tasks (Table 3) and WMC tasks (Table 4).

Table 3 Mean ratings, standard deviations, and correlations for the framing tasks

Note: Ratings ranged from 1 (definitely would favor safe option) to 7 (definitely would favor risky option).

Table 4 Means, standard deviations, and correlations for the WMC tasks

Note. Alpha coefficients in italics on the main diagonal (absolute/partial score), calculated over the recall scores in the three separate blocks (Redick et al., 2012, p. 169). Correlations between the absolute (above the main diagonal) and partial scores (below the main diagonal).

3.2. Measures’ psychometric properties

Framing problems: Internal consistency. The internal consistency of the framing problems was Cronbach’s alpha = 0.87, which is higher than the values reported in our reference studies (Berthet, 2021, Study 2; Fagley et al., 2010; Simon et al., 2004).

Working memory capacity: Internal consistency. The internal consistency of the Ospan partial score, calculated over the proportion of correctly recalled letters in the nine trials,⁶ was Cronbach’s alpha = 0.86, which is higher than the value of 0.71 reported by Oswald et al. (2015) and deemed very good (Greiff & Allen, 2018). The internal consistency of the composite WMC measure, calculated over the total partial scores across the three complex span tasks, was Cronbach’s alpha = 0.77, which is roughly the same as the value of 0.76 reported by Oswald et al. (2015) and deemed 70: good (Greiff & Allen, 2018). We used the latter measure to test Hypothesis 2.

Working memory capacity: Confirmatory factor analysis. To investigate the construct validity of the WMC measure, we conducted confirmatory factor analysis (CFA) of the number correct in each trial, assuming one factor. As multivariate normality was violated, we used robust estimators of model fit. The chi-square test indicated no exact fit, χ²(321) = 378.97, p = 0.014. However, approximate fit indices (RMSEA = 0.023, SRMR = 0.036) and incremental fit indices (CFI = 0.984, TLI = 0.983, IFI = 0.978) were excellent, indicating that the hypothesized model fit the data well (Greiff & Allen, 2018, Table 1). Given the good internal consistency of the WMC measure and the excellent fit of the CFA model, we used the full WMC measure as planned.

3.3. Confirmatory analyses

3.3.1. Analytic approach

For the close replication (Hypotheses 1a and 1b), we used linear regression predicting the ratings on the disease problem by the frame (gain/loss), the Ospan score, and the frame by Ospan interaction (Table 5). A statistically significant interaction showing a larger framing effect on higher levels of WMC was to be treated as a successful replication.

Table 5 Confirmatory analyses: Overview

Note. Model in bold is the best model according to AIC (Seedorff et al., 2019).

For the extended replication (Hypothesis 2), we used mixed-effects modeling (MELM) to predict the ratings on the framing tasks by the frame (gain/loss), the WMC score, and the frame by WMC interaction. To infer the random-effects structure of the data, we compared the three plausible models, namely a model with slopes varying by framing items, an independent slopes model excluding the correlation between by-item varying slopes and intercepts, and a model that only included random intercepts for items and participants (Table 5). We selected the best model based on AIC and AICc (Seedorff et al., 2019). Then we compared the best model to a model excluding the interaction term. A significant likelihood ratio test (Winter, 2019) favoring the model that included the interaction term would be treated as a successful replication.

To further investigate the relative evidence in favor of the effect versus the null hypothesis, we calculated BF using the brms R package (Bürkner et al., 2020). Unlike frequentist analysis, BF allows inferring not only absence of evidence but also evidence of absence. In other words, it can inform if a null result supports the null hypothesis or is due to the insensitivity of the data (Dienes, 2014). We modeled the alternative hypothesis to have a normal distribution with a mean equal to positive SESOI = 0.034 (or negative SESOI if the effect turned out to be negative) and SD = 1/2 × SESOI (Dienes, 2014). We used default priors for the remaining effects.

In all models, we centered the WMC scores and used a deviation coding for the frame (gain = −0.5, loss = +0.5). These transformations do not qualitatively change the results but reduce non-convergence issues and make it easier to interpret interactions (Brown, 2021; DeBruine & Barr, 2021; Winter, 2019).

All analyses were conducted in R (R Core Team, 2021) and the following packages: afex (Singmann et al., 2023), beepr (Bååth & Dobbyn, 2018), brms (Bürkner et al., 2020), broom (Robinson et al., 2023), broom.mixed (Bolker & Robinson, 2022), car (Fox et al., 2023), cowplot (Wilke, 2020), faux (DeBruine et al., 2023), foreign (R Core Team et al., 2023), ggeffects (Lüdecke et al., 2023), lavaan (Rosseel et al., 2020), lme4 (Bates et al., 2020), MASS (Ripley et al., 2023), MBESS (Kelley, 2021), and tidyverse (Wickham & RStudio, 2019). The R code is available at https://osf.io/ektfd/.

3.3.2. Close replication

For the close replication, we used the disease problem and the Ospan absolute score (Hypothesis 1a). As a robustness check, we also analyzed the data using the Ospan partial score (Hypothesis 1b). Table 5 displays all the confirmatory analyses using R formula notation. Table 6 displays the estimates from these models.

Table 6 Confirmatory analyses: Model estimates

Abbreviations: OLS, ordinary least squares regression; fixed effects: beta, regression coefficient (WMC has been centered, Frame has been deviation coded, i.e. gain = −.5, loss = +.5); random effects: SD, standard deviation. *p < 0.05, ^** p < 0.01, ^*** p < 0.001.

Manipulation check. To check if the framing of the disease problem was effective, we performed linear regression with ratings on the disease problem as the outcome variable and frame as a single predictor. As expected, the frame significantly predicted ratings, beta = 1.37, t = 6.98, p < 0.001, confirming that the framing manipulation was successful.

Assumption checks. We also checked the assumptions underlying multiple linear regression, namely normality, linearity, homoscedasticity, and collinearity (Winter, 2019). The normality assumption was violated in the models used to test both Hypotheses 1a and 1b.⁷ Bootstrapping the regression model (Fox & Weisberg, 2021) over 10,000 samples confirmed the results from the regression model (Table 6). To test Hypotheses 1a and 1b, we used the parameters estimated from the ordinary least squares regression.

Hypothesis 1a: Frequentist test. In the model using the Ospan absolute score, the critical interaction term, b = 0.002 (Table 6), was very close to 0, and the 95% confidence interval, [−0.03, 0.03], crossed 0, meaning no substantial change in the framing effect as WMC increases (Figure 3A). This effect is descriptively inconsistent with Corbin et al.’s (2010) and is not statistically significant, p = 0.89. Hypothesis 1a was therefore not supported.

Figure 3 Close replication: Observed (points) and predicted (lines) framing ratings as a function of WMC (A: Absolute Score, B: Partial Score).

Hypothesis 1a: Bayes factor. The BF, BF ₁₀ = 0,27, shows that the data are 3.75 times more consistent with a model omitting the interaction term than with a model including it, meaning moderate evidence in favor of the null hypothesis of no interaction over Hypothesis 1a (Stefan et al., 2019, Table 1).

Hypothesis 1b: Frequentist test. In the model using the Ospan partial score, the critical interaction term, b = 0.007 (Table 6), was very close to 0, and the 95% confidence interval, [−0.04, 0.04], crossed 0, meaning no substantial change in the framing effect as WMC increases (Figure 3B). This effect is descriptively inconsistent with Corbin et al.’s (2010) and is not statistically significant, p = 0.95. Hypothesis 1b was therefore not supported.

Hypothesis 1b: Bayes factor. The BF, BF ₁₀ = 0.32, shows that the data are 3.10 times more consistent with a model omitting the interaction term than with a model including it, meaning moderate evidence in favor of the null hypothesis of no interaction over Hypothesis 1b (Stefan et al., 2019, Table 1).

3.3.3. Extended replication

Model selection. No models showed singular fit or convergence issues. The best model in terms of AIC was the independent slope model (Table 5).

Manipulation check. To check if the framing manipulation was effective overall, we compared two mixed-effects linear models with the same outcome variable and random-effects structure as the best model. A model including the frame as a single fixed predictor fit the data better than the model without the frame (i.e., only including the fixed intercept), χ²(1) = 15.21, p < 0.001, confirming that the framing manipulation was successful.

Assumption checks. [IF all LM assumptions met:] All assumptions underlying linear models were met satisfactorily in the best model.⁸

Hypothesis 2: Test. In the best model, the critical interaction term, b = −0.004 (Table 6), was very close to 0, and the 95% confidence interval, [−0.016, 0.008], crossed 0, meaning no substantial change in the framing effect as WMC increases (Figure 4). This effect is descriptively inconsistent with Corbin et al.’s (2010). Moreover, a likelihood ratio test indicated that the model including the frame × WMC interaction did not fit the data better than a model without the interaction, χ²(1) = 0.48, p = 0.49. Hypothesis 2, that higher WMC predicts a larger framing effect more generally, was, therefore, not supported.

Figure 4 Extended replication: Observed (points) and predicted (lines) framing ratings as a function of WMC.

Hypothesis 2: Bayes factor. The BF, BF ₁₀ = 0.047, shows that the data are 21.00 times more consistent with a model omitting the interaction term than with a model including it, meaning strong evidence in favor of the null hypothesis of no interaction over Hypothesis 2 (Stefan et al., 2019, Table 1).

3.4. Exploratory analyses

To check if the problems were successful in eliciting a framing effect within subjects, we subtracted the score in the gain frame from the score in the loss frame for problems presented in both frames, and ran several mixed-effects linear models that included the difference score as the outcome variable and the frame as a single fixed predictor. The best-fitting model was the one including only random intercepts for participants and framing problems. It fit the data significantly better than a model that only included the fixed intercept, χ²(1) = 50.00, p < 0.001, confirming that the problems successfully elicited a framing effect within subjects.

We also checked whether the difference scores are related to WMC by running mixed-effects linear models that included the difference score as the outcome variable and the frame, the WMC score, and their interaction as fixed predictors. The best model, which contained only random intercepts for participants and framing problems, did not fit the data significantly better than the model omitting the interaction term, χ²(1) = 0.75, p = 0.39. In short, there was no evidence within this study that WMC is related to the within-subjects framing effect.

4. Discussion

Corbin et al. (2010) found a larger risky-choice framing effect among people with higher compared with lower WMC. This finding is unusual because most past research failed to find an association between cognitive capacity and the between-subjects framing effect (Stanovich & West, 2008). Whenever any association has been found between cognitive capacity and thinking problems more generally, it was that higher capacity was positively rather than negatively associated with normative performance. Given the potentially important theoretical implications of the association between WMC and the risky-choice framing effect, we set out to replicate and extend it in a registered report, using both the original measures and composite measures of framing and WMC.

In contrast to the original findings, no significant frame by Ospan interaction was found in predicting ratings on the disease-framing problem (Tversky & Kahneman, 1981), using either the absolute or the partial Ospan score. BFs showed moderate evidence in favor of the null hypothesis, thus corroborating the frequentist findings. The close replication was thus unsuccessful.

We were also interested in going beyond the particular tasks used by Corbin et al. (2010), so we extended the materials using six framing tasks related to human lives and three WMC tasks with different processing components (numeric, verbal, and spatial). Consistent with the close replication, the combined WMC score did not significantly interact with the frame to predict the ratings on the framing problems. The BF also showed strong evidence in favor of the null. The extended replication was thus unsuccessful.

Taken together, the present findings converge in casting doubt on the hypothesis of greater susceptibility to framing effects among individuals of higher WMC. All six preregistered tests, including three frequentist and three Bayesian ones, point in the same direction of no association. Such coherent and unequivocal empirical evidence entails a proportionally strong conclusion that the between-subjects risky-choice framing effect is not associated with WMC.

In light of the present unequivocal evidence, we see two main reasons for the positive association originally found by Corbin et al. (2010). Most probably, this was a false positive result, which is common to underpowered studies (e.g., Chambers, 2017). Alternatively, the positive findings might reflect a genuine effect which is, however, due to some third factor that was not accounted for in the original study. Recall that Corbin (2013) only replicated the interaction effect among participants of low numeracy, which might also be the case for Corbin et al.’s (2010) sample. Such an intriguing three-way interaction between WMC, numeracy, and the framing effect might be theoretically defendable (e.g., from the point of view of the fuzzy-trace theory), but testing it would require a sample roughly four times as big as in the present study (Simonsohn, 2014), that is, about 1,600 participants. Until a good rationale and resources are found to test this possibility, we will content ourselves with concluding that WMC is not related, overall, to the between-subjects risky-choice framing effect.

4.1. Our findings in the context of extant empirical evidence

Our findings are consistent with previous research that failed to find association between cognitive capacity and performance on a host of decision-making tasks administered between-subjects, including framing problems (Stanovich & West, 2008). Our findings diverge from evidence for an advantage of higher WMC in decision-making tasks (Burgoyne et al., 2023; Cokely & Kelley, 2009; Del Missier et al., 2013; Dougherty & Hunter, 2003; Starcke et al., 2011). There are many differences between the problems used in those studies and the framing problems we used that could account for this divergence. Most notably, superior performance associated with higher WMC appears in tasks that demonstrate an individual’s reasoning capacity to adhere to rational norms. In framing tasks presented between subjects, on the other hand, there is neither a clear procedure nor domain-specific knowledge that the individual problem solver could follow in order to adhere to the rational norm because the latter is only tested at the supra-individual level, as coherence of people’s beliefs and preferences. This distinction between reasoning rationality and coherence rationality is important in many respects and the former does not necessarily imply the latter (Kahneman, 2000). Better access to highly automatized procedures, which is supposedly at the core of the WMC advantage, is of no use in the absence of clear rules to be followed. Therefore, even high-WMC individuals are expected to fail the stricter coherence-rationality test even though they are capable of passing less strict reasoning-rationality tests.

Our exploratory analyses also failed to find an association between WMC and the within-subjects framing effect we found, while a negative association has sometimes emerged in previous research (Bruine de Bruin et al., 2007), although not always (Del Missier et al., 2012; Stanovich & West, 1998; Toplak et al., 2014). Our primary goal behind presenting some tasks in the alternative frame was to check if the problems successfully elicited the framing effect, which they did. We were quite explicit about this aim in our instruction to participants. By contrast, previous research has typically introduced the alternative frame without an extra explanation, using all the problems from the original frame, and with a certain time interval between the two frames filled with other tasks. Given these differences, we do not consider our results to be a genuine test of a within-subjects framing effect. Consequently, we could not rigorously test for an association between WMC and the within-subjects framing effect, although we think that such a test would further advance the evaluation of the competing accounts of the framing effect.

4.2. Theoretical implications

The outcome of this study turned out to be both the most expected one empirically and also the least informative one theoretically. Four out of the six accounts we considered were compatible with the present outcome of no association between WMC and the between-subjects framing effect, while the two alternative outcomes (positive and negative association) had only two supporting accounts each (Table 1). Still, we can conclude that the findings are incompatible with two accounts, the parallel-competitive dual-process account (Sloman, 1996) and the intensified shift account (Delaney & Sahakyan, 2007). Further, we may want to assign more credit to accounts that specifically predicted no association as the single compatible outcome than to accounts that were also compatible with alternative outcomes. By this logic, the more specific default-interventionist (Kahneman, 2003; Stanovich & West, 2008) and pragmatic inference (Mandel & Kapler, 2018) accounts predicted the outcome better than the vaguer fuzzy-trace (Corbin et al., 2015) and the hybrid dual-process (De Neys & Pennycook, 2019) accounts.

To be sure, some might not agree with our interpretations of the various theoretical predictions, which are in some cases our best inferences rather than claims explicitly made by the authors of the respective theories. Such disagreement about what theories entail is a common problem in psychology where theories are not axiomatized, but it is the theories that are to blame rather than those who tested them (Evans, 2016). The burden is then on the authors to further elaborate their theories to the point that they make clear predictions that could be logically derived and empirically tested by independent researchers.

4.3. Limitations and avenues for future research

Our study had several methodological limitations. First, we used only risky-choice framing problems related to human lives. This might be the reason why the internal consistency of the framing problems was higher than usual. However, research has shown that the risky-choice framing effect is sometimes moderated by the task contents (Fagley & Miller, 1997). Future research might test whether the present findings generalize to risky-choice problems in the financial domain or to other types of framing effects such as the attribute framing effect (Levin et al., 1998).

Second, multiple definitions and measures of working memory exist (Cowan, 2017; Oberauer et al., 2018), so the association between WMC and the framing effect can be further tested using other measures of WMC. Third, the study was conducted online and thus lacked rigorous procedural control. Yet, our quality checks and descriptive statistics did not point to anything that would noticeably affect participants’ performance, so we believe the upside of online data collection, in terms of fast and easy recruitment of participants beyond student samples, outweighs its downside in this case.

Since our primary goal was to replicate and extend Corbin et al.’s (2010) findings, we missed opportunities to pit against each other the two dominant accounts of the present findings, the default-interventionist dual-process account and the pragmatic inferences account. For instance, we considered adding the word ‘exactly’ to the numbers of people affected (e.g., ‘exactly 200 people will be saved’). According to the pragmatic inference account, adding ‘exactly’ would prevent people from interpreting the number as lower bound (‘at least’) and will thus make the framing effect disappear, but according to the default-interventionist account, this single-word addition would not affect the framing effect. However, such an addition would also impede close replication. Hence, we refrained from changing the original wording of the problems. Likewise, we have refrained from testing for a WMC by framing interaction in a within-subjects design, which would be another way to distinguish between the two accounts.⁹ We look forward to future research aimed at assessing alternative explanations through refined methods. The evidence we have presented can be combined with forthcoming findings to better distinguish between the various theories of framing and its moderating factors.