Risky choice framing by experience: A methodological note

Abstract

In classic research on judgment and decision making under risk, risk is described by providing participants with the respective outcomes and probabilities in a summary format. Recent research has introduced a different paradigm – decisions-by-experience – where participants learn about risk by sampling from the outcomes, rather than by summary descriptions. This latter research reports a description-experience gap, indicating that some of the classic patterns of risk attitude reverse when people experience the risk. Recent research has attempted to investigate risky choice framing in the decisions-by-experience paradigm. I discuss how this research runs into problems in properly manipulating framing in decisions by experience. Drawing from framing research with animals, I argue that framing effects also exist in experience tasks. The classic Asian Disease task, however, awaits proper translation into an experience paradigm.

1 Introduction

Does a glass half-full contain the same amount of liquid than a glass half-empty? Is ground beef 80% lean better than ground beef 20% fat? These are examples of differently framed situations. That is, by way of describing a situation differently, as filled or empty, lean or fat, respectively, a factual difference is implied. Of course, the difference is superficial rather than substantial, existing only in description. Indeed, the factual state of affairs is identical, pertaining to the same reality. Understanding the causes and consequences of different descriptions is the goal of framing research.

A predictable and systematic effect of different descriptions of equivalent choice situations is called a framing effect. Framing effects like that just described are robust and reliable empirical findings, documented in hundreds of papers, and summarized in overviews and meta-analyses (e.g., Keren, 2011; Kühberger, 1998; Kühberger, Schulte-Mecklenbeck & Perner, 1999; Levin, Schneider & Gaeth, 1998; Mandel & Vartanian, 2011; Steiger & Kühberger, 2018). Framing effects are frequently seen as exemplary violation of the basic normative principle of invariance (or extensionality), which holds that preference ought to depend on the options, and not on their description: “preference between options should be independent of their description” (Tversky & Kahneman, 1986, p. 253).

A widely accepted typology of different tasks showing framing effects was proposed by Levin, Schneider and Gaeth (1998). The authors distinguished among risky choice framing, attribute framing, and goal framing. The paradigmatic task of risky choice framing is the Asian Disease Problem (ADP), as famously formulated by Tversky and Kahneman (1981). Here it is, stated in positive terms (lives saved):

Imagine that the US is preparing for the outbreak of an unusual disease, which is expected to kill 600 people. Two alternative programs to combat the disease have been proposed. Assume that the exact scientific estimate of the consequences of the programs are as follows:

If Program A is adopted, 200 people will be saved.

If Program B is adopted, there is 1/3 probability that 600 people will be saved, and 2/3 probability that no people will be saved.

Which program would you choose?

In the negative frame the same cover story, but a different description of the options is provided:

If Program C is adopted 400 people will die.

If Program D is adopted there is 1/3 probability that nobody will die, and 2/3 probability that 600 people will die.

Thus, participants have to decide between two identical options that are described as if they were different, namely as involving saved, or lost, lives. Framing varies the description of the choice situation, but not its outcome. The typical finding is that positive framing leads to risk aversion (a majority of experimental participants prefers saving 200 people for sure, over saving 600 people with probability 1/3 and saving none with probability 2/3), while negative framing leads to risk-seeking (the majority preferring losing 600 people with probability 2/3 and losing none with probability 1/3, over losing 400 people). This effect has attracted a lot of attention as an exemplary case of human irrationality not only in psychology, but also in economics, philosophy, and linguistics.

In an important paper, Hertwig et al. (2004) introduced a basic distinction of two decision situations. In one class of decisions, people enjoy convenient — often numerical — descriptions of the risky options, in terms of frequencies and probabilities of possible outcomes. This is the class of description-based decisions, characterized by probabilities and payoffs described in terms of summary measures (e.g., 50% chance to win $50). In the second class of risky decisions however, people do not have statistical summary data at hand. For instance, when crossing a street, or going on a date, people usually do not have available a convenient summary description of the risk of the possible outcomes. Rather, all they can do is relying on their own experience. This class of situations is called experience-based decisions by Hertwig et al. (2004). In short, in decisions-from-description people get a written or graphical description of a situation, with value and risk information being numerically available and nicely summed up. In decisions-from-experience, people either need to consult their memory for instances, or to actively sample information from the environment.

The initial explanation of the different choices in description and experience — the description-experience gap — was that low-probability events tend to be overweighted in description because their probabilities are explicitly presented. In contrast, in experience-based tasks, low-probability events tend to be underweighted, because rare events are underrepresented in small samples, in line with the skewed form of the binomial distribution for extreme probabilities (Hertwig & Erev, 2009; Hertwig et al., 2004; Hadar & Fox, 2009). This interpretation was revised, however, as research that did not use rare events also found the description-experience gap (Ludvig & Spetch, 2011). Recent interpretations of the description-experience gap therefore are more general, arguing, for example, that the experience format reduces the sensitivity to probabilities while increasing the sensitivity to outcomes (Kellen, Pachur & Hertwig, 2016). Note also that the differences are highly task sensitive. For example a reversed gap is found if full gambles are used instead of ADP-type tasks that compare a sure and a risky option (for a recent meta-analysis see Wulff, Mergenthaler-Canseco & Hertwig, 2018).

2 Experience-Based Risky Choice Framing

Here I ask whether the description-experience gap also exists for risky choice framing. The literature usually provides an answer. However, I found only a few papers on risky choice framing by experience. Specifically, we know of only three studies investigating the classic ADP in an experiential, rather than descriptive, format (Fu, Yu, Ni & Li, 2018; Gonzalez & Mehlhorn, 2016; Vallée-Tourangeau, Vallée-Tourangeau & Ramasubramanian, 2016). Gonzalez and Mehlhorn (2016) presumably introduced the basic procedure (see their Figure 1; see also the practically identical Figure 2 in Vallée-Tourangeau et al., 2016; or Figure 1 in Fu et al., 2018). In the words of Gonzalez and Mehlhorn (2016): “Participants are presented with the ADP scenario and are told to sample the outcomes of the two programs for as long as they want and in whatever order they want, by pressing the respective buttons (labelled A and B). Participants do not know which is a risky and a safe option […]. When a button is selected, an outcome is drawn from the respective distribution and is displayed” (p. 1167). Fu et al. (2018) used the same instruction. The third study used a slightly different manipulation of a life threatening risky choice framing task (see Figure 2 in Vallée-Tourangeau et al., 2016): “participants were presented with two packs of 25 cards and asked to assume that the possible consequences of the programs were represented by the two packs of cards in front of them. They were informed that, upon turning a card over, they would see a possible consequence for the selected program in terms of the number of people saved [killed]” (p. 2629).

The results of the three studies are inconsistent: Gonzalez and Mehlhorn (2016) report that the framing effect disappeared in the experience condition. However, a closer look at their findings (especially the difference between the description and the experience conditions) indicate that the effect is reduced, while it is still in the expected direction. Fu et al. (2018) report that the effect disappeared only in the loss condition but found no change for the gain condition. The combined data over three experiments however, indicate that the effect persists in the experience conditions, though somewhat smaller than in description. Vallée-Tourangeau et al. (2016) report entirely contrasting findings: the effect reversed for both framing conditions. Taken together, the issue still is contested.

These experience-based manipulations have in common that the outcomes of either of the two options are experienced repeatedly (as often as participants sample from the deck of cards). This is the traditional sampling paradigm of decisions-by-experience, where people repeatedly sample, and express their preference only at the end of the sampling process. In the case of the ADP, only three outcomes are possible in either framing condition:

Gain condition:

• • 200 people saved (sure gain)

• • 600 people saved (risky gain)

• • 0 people saved (risky gain)

Loss condition:

• • 400 people dead (sure loss)

• • 0 people dead (risky loss)

• • 600 people dead (risky loss).

For example, in the positive framing condition, a participants might see the following series:

• • 200 people saved (by clicking Option A)

• • 0 people saved (by clicking Option B)

• • 0 people saved (by clicking Option B)

• • 200 people saved (by clicking Option A)

• • 600 people saved (by clicking Option B)

• • 200 people saved (by clicking Option A).

Outcomes would appear according to their probabilities, that is, when clicking Option A one is always seeing “200 people saved”, and when clicking Option B one is seeing either “0 people saved” (with 2/3 probability), or “600 people saved” (with 1/3 probability).

OK, but what is experienced here? The answer resolves around what the respective outcome distribution is. The outcome distribution is defined over the number of people being saved/dying; it is NOT the distribution of descriptions of these outcomes. Thus, participants ought to experience single outcomes as a result of their sampling (e.g., whether someone is dying or not dying), rather than descriptions of summary outcomes.

I argue that the sampling procedure used in the experiments described above misrepresents the situation, by implying that the ADP situation would repeatedly exist in identical form, for instance in different countries. However, this is not the intended meaning. Rather, the disease situation exists only once, offering different outcomes for 600 affected individuals (in the US). A series of samples on single outcomes might be, for instance:

• • person #1: saved

• • person #2: not saved

• • person #3: not saved

• • person #4: saved

• • person #5: saved

• • person #6: saved.

To repeat: the ADP is referring to a specific situation where different people each are surviving, or dying. To put it differently: in the classic ADP, the underlying gamble consists of a finite number of 600 draws without replacement from the stake of 600 events (lives). The risky events are single lives saved/not saved. The ADP, as it is manipulated in the studies described above, consists of an infinite number of identical draws of summary descriptions of outcomes (e.g., 200 people saved), varying in wording. This manipulation misrepresents the structure of the ADP.

Importantly, in the description format the framing is done by the language used for describing the options (i.e., save, die). The cover-story, that 600 people are in danger, reassures equivalence of options, but has no effect on the reference point. Rather, the cover-story could say that 300 people are in danger, or that 6000 people are, without having any effect on the framing: 200 people saved would still be a gain, and 400 people dying would still be a loss. That is, the cover-story is decoration in framing by description.

In the experience format, proper manipulation of framing requires — in addition to the sampling from the options — the experiential manipulation of the reference point. We are aware of only one study that did this with human participants. Mishra, Gregson and Lalumiere (2012) crossed framing and description-experience formats in a lottery task. Framing was manipulated by providing an initial endowment of nothing and choices between sure or risky gains (positive frame), or an initial endowment equivalent to the maximum possible amount of earnings and choices between sure or risky losses (negative frame). A standard framing effect appeared in both formats; i.e., there was no description-experience gap.

The animal literature on framing is an important source of inspiration for experiential framing. Research with animals is possible ONLY in the experiential format, and researchers have come up with ingenious solutions to accomplish this. Marsh and Kacelnik (2002) were among the first to describe a framing experiment on starlings. Each of 14 starlings was habituated in standard trials to expect either 1 (gains condition), or 7 pellets (loss condition) after pecking on a key. In choice trials, the animals could choose between either (a) always obtaining 4 pellets, or (b) 2 or 6 pellets of food with equal probability. These options are equivalent in terms of outcome, but one option delivers for sure, while the other is risky. Importantly, the manipulation of the reference point (i.e., the framing) was done in the standard trials: If starlings expected 1 pellet in standard trials, the choice between either getting 4 pellets for sure, or opting for a chance to get 2 or 6 pellets is between two gains, as both options deliver more pellets than does the standard. In contrast, if starlings expected 7 pellets in standard trials, the identical outcomes (either getting 4 pellets for sure, or opting for a chance to get 2 or 6 pellets) are losses; starlings get less than in standard trials. Although there is discussion on whether this indeed requires the animals to frame outcomes (Houston & Wiesner, 2020), this method appears to be possible way to investigate framing by experience. The frame is manipulated by habituating the animals to initial outcomes that are either low, such that the (higher) choice options are gains, or high, such that the identical (lower) choice options are losses. The domain is always food choice. Marsh and Kacelnik (2002) did their study with starlings; Krupenye, Rosati and Hare (2015) did it with bonobos and chimpanzees; Lakshminarayanan, Chen and Santos (2011) did it with capuchin monkeys; and Bhatti et al. (2014) did it with rats. In all these studies the animals exhibited reference-dependent behavior in a way similar to human preferences in the classic framing paradigm. Taken together, it appears that there is the usual framing effect, i.e., no description-experience gap (although the description condition could not be tested), when testing framing in experience. Note the important difference: the framing of animals depends entirely on the reference point (by changing the standard outcome), while with humans the framing depends entirely on the description of the options.

Proper framing by experience runs into another problem, however: the task involves a safe option, but participants in the experience condition — be it animals or humans — can never be sure that their homogeneous drawings pertain to a safe option. Glöckner et al. (2016) convincingly showed that this information asymmetry is partly responsible for the description-experience gap. In their meta-analysis, Wulff et al. (2018) correspondingly found that the size of the descriptive-experience gap was largest for problems involving a risky and a safe option, pointing to the possibility that the safe option actually is not considered “safe” in experience. Gonzalez and Mehlhorn (2016) report that participants under-explored in their study, by drawing only about 7 samples before choosing. That is, if participants divided the drawings about half between the two options (what they did), they saw the sure outcome 3–4 times (see O’Brien, (2020) for different evidence on under-exploration). Interestingly, this amount of sampling is much lower than reported by Wulff et al’s. (2018), who reported an average of 22 draws for problems involving a risky and a safe option. Presumably, the value of information — and thus the amount of sampling — depends on the size of the stimuli sampled from. Take the choice between an option (A) offering $4, p=0.8, $0, p=0.2, and option (B) offering $3, p=1. Participants here would sample events like 4, 4, 0, 4, 0, 4, 4, …(Option A), or 3, 3, 3, 3, 3, …(Option B). Samples from option A would be inhomogeneous, while samples from option B would be homogeneous. Now, what would participants experience in a properly manipulated ADP? They would see only single cases (e.g., alive, alive, alive, not alive, …) in either option. That is, homogeneity is not an issue here as participants are seeing only one outcome at a time. In sum, severely limited sampling in experience here has various consequences: (i) it renders the distinction between sure and risky option pointless; (ii) it renders the notion of homogeneity inappropriate; and (iii) options become non-equivalent in terms of expected value, since equivalency needs not show up in small samples.

Let us imagine doing a proper ADP experience framing task, where participants are sampling individual outcomes from the population of 600 cases. How could a participant evaluate the riskiness of options? She could not, since there is no difference between options in experience: each single draw brings, with the same probability, an outcome where the person is alive or not (dead or not). Experientially it is impossible to distinguish between levels of risk (i.e., sure vs. risky). Indeed, the options are identical not only with respect to expected value, but also with respect to risk: the option of saving 200 people out of 600 for sure can be rewritten as saving all 600 people with 1/3 probability or saving none with 2/3 probability. The impression of a sure option only exists because the risk part is hidden. In short, in an operationalization of the ADP by experience the concept of risk collapses and a characterization of options as sure or risky does not make sense.

To reiterate: imagine you are participating in a classic risky choice framing study by description, receiving the following option “200 people will be saved”. If you had to decide on the experimental condition that this represents, what would you say? The sure or the risky option, in gain or loss framing? The answer is obvious: sure option, gain frame. Now imagine participating in the appropriate experience version of the study. You draw a sample of individual outcomes, containing, say, five events: alive, dead, dead, dead, alive. Again you are asked: which condition are you in, sure or risky, gain or loss framing? The question is impossible to answer, since there is only one situation, and the same draw can pertain to either the sure or risky option, and gain or loss framing. Of course, framing is possible (e.g., for the example above a positive framing with negation would be: alive, not alive, not alive, not alive, alive) by labelling the focal outcome as saving lives and using negation, or as losing lives and using negation. Such a framing manipulation is unrelated to risk, however, and thus the essence of risky choice framing is lost, reducing the risky choice framing task to an attribute framing task. Even after exhaustive sampling of 600 draws a distinction between sure and risky option is meaningless: either option ends up with a proportion of 200/400. In sum, in the experience format, there neither is a way to measure something like the attitude toward risk, nor a manipulation of the reference point due to the framing.

As I have argued at length, the studies purporting to manipulate risky choice framing in experience (Fu et al., 2018; Gonzales & Mehlhorn, 2016; Vallee-Tourangeau et al., 2016) have participants experience different summary descriptions, rather than different individual outcomes. This, by itself, is an interesting manipulation, but it does not frame the basic experience. It appears that proper framing by experience is possible, but requires specific procedures. Most importantly, the reference situation is a central ingredient of the procedure. Deviations from the reference point define the framing. In contrast, manipulating the frame linguistically runs into difficulties when attempting to manipulate framing in experience. Indeed, this state of affairs may be just indicative of the essence of framing: to describe identical experiences differently. That the sum of the experiences can be described differently does not necessarily entail that the individual experiences are different.

3 Conclusion

We have discussed the problems of risky choice framing in experience. Some studies have attempted to manipulate framing in experience, but these fail in rendering the proper experiences available, when modelled after the ADP. This is due to the fact that, when the problem is properly modeled at the level of individuals (dying vs. saved), both options are indistinguishable at the observational level (i.e., in both programs one experiences 1/3 saved and 2/3 dying).

Animal studies cannot rely on summary descriptions and thus have used an ingenious way to manipulate framing by direct changes of outcomes from a reference point. These studies found human-like framing effects in food choice tasks for all species used. I am not aware of (experiential) food choice risky framing tasks with human participants. Such tasks might actually be useful in deciding on the status of the risky choice framing effect in experience. For the time being, the evidence for framing effects in experience with humans is sparse, but our expectation is that there will not be a description-experience gap here. It is hard to see, however, how the classic ADP could possibly be translated into the experience format without losing its central ingredients: the distinction between sure and risky option, and the identification of the valence of the outcomes, namely gain or loss.