Cognitive reflection as a predictor of susceptibility to behavioral anomalies

Abstract

To study the effect of cognitive reflection on behavioral anomalies, we used the cognitive reflection test to measure cognitive reflection. The study was conducted on 395 Iranian university students and shows that subjects with lower cognitive reflection are significantly more likely to exhibit the conjunction fallacy, illusion of control, overconfidence, base rate fallacy, and conservatism. In addition, test scores are correlated with risk preferences. The results do not show any relationship between cognitive reflection and self-serving bias or status quo bias. We also find that gender is significantly related to illusion of control and self-serving bias.

1 Introduction

After Frederick (2005) introduced the Cognitive Reflection Test (CRT), a number of researchers studied the relationship between cognitive reflection and behavioral biases. Frederick (2005) shows that individuals with high CRT scores are generally more willing to be patient and willing to gamble in domain of gains; Benjamin, Brown and Shapiro (2006) and Slonim, Carlson and Bettinger (2007) found similar relationships. Oechssler, Roider and Schmitz (2009) found that subjects with low CRT scores exhibited the conjunction fallacy and conservatism more often than subjects with high CRT scores. They specifically posit that “…people with higher cognitive abilities might save more and receive higher expected returns; potentially leading them to play a more pronounced role in financial markets than subjects with lower cognitive abilities”. Hoppe and Kusterer (2011) find that CRT is predictive of susceptibility to the base rate fallacy and conservatism. They postulate that CRT is instrumental only when there is a correct solution for the bias through analytical calculations. Moreover, Hoppe and Kusterer report that the endowment effect is not correlated with CRT scores. Albaity, Rahman and Shahidul (2014) also replicate the relationship between cognitive reflection and behavioral biases with Malaysian subjects. The main finding of this literature is that cognitive reflection is predictive of susceptibility to behavioral anomalies, though cognitive reflection is not the only predictor.

This study investigates the relationship between cognitive reflection and behavioral anomalies in Iran. It includes the status quo bias, illusion of control, and self-serving bias. We use the CRT (Frederick, 2005) as a measure of cognitive reflection, as it is hypothesized to measure the System 2 decision-making abilities in dual process theory. This theory differentiates between two types of thinking: System 1 thinking or intuition, which corresponds to intuitive judgments that are fast and automatic, and System 2 thinking or reasoning, which applies to the class of problems which require reasoning and are subject to conscious judgments (Kahneman, 2003). The current study replicates the earlier studies and their main findings regarding the conjunction fallacy (Hoppe & Kusterer, 2011; Oechssler et al., 2009), overconfidence (Hoppe & Kusterer, 2011), the base rate fallacy (Hoppe & Kusterer, 2011), conservatism (Albaity et al., 2014; Hoppe & Kusterer, 2011; Oechssler et al., 2009), and risk preferences (Frederick, 2005; Oechssler et al., 2009). The results show that these biases are significantly related to CRT scores. Regarding overconfidence, subjects with higher CRT scores have a significantly more precise self-assessment (Hoppe & Kusterer, 2011), and subjects with lower test scores tend to be significantly overconfident. There is no evidence that self-serving bias and CRT scores are correlated. Further, CRT scores do not indicate any relationship with time preference and the status quo bias.

This paper introduces a new topic, which is the relationship between gender and behavioral anomalies. The results show that female subjects are significantly more likely to exhibit illusion of control and the self-serving bias in decision making contexts.

2 Method

Four hundred and ten university students from three universities (Allameh Tabataba’i University, Amirkabir University of Technology, and Sharif University of Technology) participated in the research and completed a questionnaire (15 students did not complete the questionnaire, so the results are based on 395 subjects). I distributed a questionnaire with the CRT and tests of cognitive biases. (The Appendix provides an English translation.) The data were collected during February to June, 2015. The average age of the subjects was 22.4 (SD=3.62), and 63% of subjects were male (249 subjects). Additionally, 66.3% were bachelor’s students, 29.6% master’s students, and the remaining 4.1% were doctoral students. Subjects who correctly answered 2 or 3 CRT questions were paid 30 thousand rials, with 259 subjects receiving this payment.

2.1 Cognitive reflection test

The CRT (Frederick, 2005) is a three-item test with two types of answers for each question: a reflective answer and an impulsive answer. The CRT includes the following questions.

• • A bat and a ball together cost 110 cents. The bat costs 100 cents more than the ball. How much does the ball cost? (impulsive answer: 10 cents; reflective answer: 5 cents).

• • If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets? (impulsive answer: 100 min; reflective answer: 5 min).

• • In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake? (impulsive answer: 24 days; reflective answer: 47 days).

The average number of correct CRT responses (in this study) is 1.89, which ranks our subjects among MIT and Princeton subjects based on Frederick’s (2005) sample. In our sample, 41.3% of subjects answered all three questions correctly, 24.3% answered two questions correctly, 17.5% answered only one question correctly, and 17% did not answer any questions correctly (see Table 1).

Table 1: CRT scores, by location.

Table 2 shows the distribution of answers separately for each question. Male subjects received higher average test scores (2.06) than female subjects (1.06; p < 0.001, MWU test), which is similar to the findings in Frederick (2005), Hoppe and Kusterer (2011), and Oechssler et al. (2009).

Table 2: Distribution of answers to CRT questions

3 Results

We divide subjects into two groups based on their CRT score, with the “high” group answering 2 or 3 questions correctly (259 subjects) and the “low” group answering 1 or 0 questions correctly (136 subjects).¹ The high group is regarded as reflective decision makers and the low group as impulsive decision makers. The summary of the results by these two groups is shown in Table 3.

Table 3: Behavioral biases and CRT groups. To test the relationship between CRT scores and two behavioral anomalies, self-serving bias and base rate fallacy, the Mann-Whitney U test is used, and for other behavioral anomalies, Fisher’s exact test (2-sided). The text reports the relationship between the CRT score (as a continuous measure) and each of the behavioral anomalies.

3.1 Risk preferences

We asked subjects two questions related to risky choice. In question 1, which is in the domain of gains, they have to choose between a sure payment of x, and a lottery with a 75% probability of getting 2x and 25% of getting 0. In question 2, which is in the domain of losses, they have to choose between paying –x or a lottery with a 75% probability of paying –2x.

On question 1 (gains), 49.4% of the high group and 72.8% of the low group chose the sure payment (p<0.001, Fisher test). On question 2 (losses), 88.4% of the high group and 80.9% of low group chose the sure payment (p=0.049). The CRT score (0–3) correlated negatively with risk aversion in gains (p<.001, based on Kendall’s τ) and positively with risk aversion in losses (p=.019). In this study 44.8% of the high group and 58.8% of the low group chose the sure payment in both gain and loss domains, thus showing higher risk aversion as compared to Oechssler et al. (2009).

3.2 Time preference

Subjects were asked about their preference in choosing to receive a prize now or after a month with 5% interest. We found that 58.3% of the high group and 50% of the low group chose to wait (were patient). This reflects a non-significant relationship between time preference and cognitive reflection in our study, which conflicts with previous studies’ (Benjamin et al., 2006; Frederick, 2005; Slonim et al., 2007) findings that subjects with high CRT scores are more patient. (The result was also non-significant using the CRT score.)

3.3 Conjunction fallacy

Subjects were asked the “Linda” question from Tversky and Kahneman (1983): “Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.” Then, they were asked to indicate which of the following statements is more likely to be true: (1) Linda is a bank teller. (2) Linda is a bank teller and is active in the feminist movement”.

49.4% of the high group and 69.1% of the low group choose the biased answer (Linda is a bank teller and is active in the feminist movement), with this group difference being significant based on a Fisher exact test (p<.001; and the results was also significant at p<.001 for the Kendall correlation with the CRT score). Tversky and Kahneman (1983) found that 85% of the subjects showed the conjunction fallacy, while in our sample, 56.2% of all subjects showed it. However, the experiments are not comparable because Tversky and Kahneman provided more options, thus disguising the critical comparison.

3.4 Self-serving bias

The self-serving bias is the belief that individuals tend to ascribe success to their own abilities and efforts while simultaneously ascribing failure to external factors (Campbell & Sedikides, 1999). To study the self-serving bias, we use a reduced and modified version of the attributional style questionnaire from Dykema, Bergbower, Doctora and Peterson (1996). Responses are reported on a 10-point scale, where 0 means “the problem or situation has occurred solely by myself” and 10 means “the problem or situation has occurred solely by circumstances”. In each of the questions, subjects encounter a problem or a bad event, and they decide whether the cause of the problem is internal or external. We use the scale to study differences between the two CRT groups in attributing problems to internal or external factors.

The average score of the three questions is used as the measure of susceptibility to self-serving bias. The questions are as follows:

1. 1. When you feel sick and tired most of the time.

2. 2. When you have financial problems.

3. 3. When you have a serious argument with someone in your family.

Although the average score of the high CRT group (4.84) is lower than the average score of low CRT group (4.92), the difference is not statistically significant.

3.5 Status quo bias

In choosing among alternatives, individuals display a bias toward maintaining the status quo alternative (Samuelson & Zeckhauser, 1988). To study the bias, we use a slightly modified question based on Samuelson and Zeckhauser (1988). Subjects were presented with the following scenario: “You are a serious follower of the financial news. Recently you realized that you inherited 10,000 shares of an auto company from your uncle. You are deliberating whether to leave the stocks intact or to change it by investing in other securities. Your options are to:

1. 1. Hold the stocks, which have an expected return of 28 percent.

2. 2. Invest in a pharmaceutical company stock which has a 50% chance of doubling in value and a 50% chance of declining 30% in value.

3. 3. Invest in a government bond that will yield a tax-free return of 24%.”

Individuals who choose option 1 show the status quo bias. Of our subjects, 32.8% of the high group and 48.5% chose this option. However, neither this difference nor the correlation betwen CRT score and the bias is significant.

3.6 Illusion of control

Langer (1975) stated that “…people behave as if they have control over [an] uncontrollable event even when the fact that success or failure depends on chance is salient”. This behavior is called the illusion of control. To study it, we use the following question: “In playing games that use dice, which condition would you prefer? (1) I throw the dice myself, (2) it makes no difference whether others throw the dice for me, or I do it myself”. Subjects in the high CRT group are considerably less susceptible to this bias. 62.2% of the high group and 83.8% of the low group show the bias (p<0.001, Fisher test; also p<.001 by Kendall correlation with CRT score). Overall, 69.6% of all subjects exhibited illusion of control on this question.

3.7 Overconfidence

Overconfidence involves people’s inclination to overestimate their own abilities to successfully perform a particular task (Brenner, Koehler, Liberman & Tversky, 1996). To study overconfidence, we asked subjects five questions related to general knowledge. After answering the questions, subjects were asked to estimate the number of correct answers.² For each CRT group, we divided the subjects into three subgroups: those who were overconfident, those who had correct self-assessments, and those who were underconfident. Overconfident (underconfident) refers to those whose estimated number of correct answers was more (less) than the actual number of correct answers, and correct self-assessments represent those whose reported number of correct answers was equal to the actual number of correct answers.

As shown in Table 3, subjects in the low CRT group tend to overestimate the number of correct answers significantly more often than subjects in the high CRT group. That is, 66.2% of the low group and 47.1% of the high group exhibit overconfidence (p<0.001, Fisher test). Compared to the low CRT group (16.9%), a larger proportion of subjects in the high CRT group (34.4%) correctly estimate the number of correct answers (p < 0.001, Fisher test). Overconfidence is thus more prevalent in subjects with low cognitive reflection. However, this result is due entirely to the correlation between CRT score and actual number correct (r = 0.31, p<.001), plus the absence of any correlation between CRT score and confidence (judged number correct, r = 0.02). When confidence is regressed on both CRT.score and actual number correct, it is highly influence by number correct (coefficient 0.24, p<.001), but the negative effect of CRT.score is not significant (coefficient –0.06, p=0.207), as it could be if reflection actually reduced overconfidence. It is also worthy of note that, in many previous studies, people were more overconfident when facing acculturated test (like the one we used) than in sensory and perceptual tasks (where they often exhibited underconfidence). Thus, the overconfidence found in this study may depend on the use of an acculturated test.

3.8 Base rate fallacy

The base rate fallacy refers to a tendency to ignore statistical information of prior probabilities in favor of the specific evidence concerning an individual case (Kahneman & Tversky, 1973). To examine the base rate fallacy, we use a modified question based on the paper by Tversky and Kahneman (1974). Subjects answered the following question: “There are 100 officers in a commercial building, including 70 lawyers and 30 engineers. We interviewed a person from the building. Dick is a 30-year-old man. He is married with no children, a man of high ability and motivation, he promises to be quite successful in his field. He is well liked by his colleagues. What is the probability that he is an engineer?”. Our correct answer is 30%. The high CRT group reported an average guess of 37.4% which is significantly lower than the 40% from the low CRT group (p<0.025, Mann-Whitney U test; also p=.002 when tested with a Kendall correlation with the CRT score). The high group was more likely to give the correct answer (68.0% vs. 51.5%) and less likely to give the most common incorrect answer, 50%, which ignores base rates completely (7.7% vs. 24.3%).

3.9 Conservatism

Conservatism refers to behavior where one is slow to change beliefs in the face of new evidence. For example, investors subject to conservatism might disregard the information in a company’s earnings (or some other public) announcement. This might happen because the investor believes that the information is true only temporarily, or it might happen because the individual clings to his/her prior estimates of earnings (Barberis, Shleifer & Vishny, 1998). Therefore, the investor might inadequately adjust his/her valuation of shares in response to the announcement. To study this bias, we used the following question: “You have a friend who works in a brokerage firm, and he is active in stock trades. Recently he suggested to you to buy a pharmaceutical company’s stock, and he said that, these days, he receives numerous buy order for that stock. Furthermore, you found other information which supports the company’s profitability. Based on these facts, you bought 20,000 shares yesterday. Today, you heard news about fire in one of the company’s warehouses, news that your friend confirmed. What is your reaction? a) I wait for later news to get assured about the events that happened in the company, and after that I will decide, b) I send sell order to my broker immediately.”

In the study, 64.9% of the high CRT group and 84.5% of the low CRT group exhibited conservatism. This difference is significant at p<0.001 (Fisher test; also significant at p<.001 by a Kendall correlation with CRT score). In all, 71.6% of subjects in the study exhibited conservatism, though the bias is less prevalent in the high group. It is worth noting that, in many previous studies, the measure of conservatism was the original measure that Edwards (1968) used which involved numerical probabilities. Conversely, in this study, the measure is more related to perception (or intuition).

4 Gender and behavioral anomalies

The results show that two biases, illusion of control and the self-serving bias, are correlated with gender. As shown in Table 4, female subjects tend to attribute problems to external factors more often, as compared to male subjects. The average score for female subjects is 5.15, and the average score for male subjects is 4.72 (p<0.005, U test). Further, female subjects exhibit an illusion of control more often than male subjects (76.7% vs. 65.5%; p<0.023, Fisher test). We did not find any other gender differences in the biases that were measured.

Table 4: Behavioral anomalies by gender.

5 Conclusion

This study revealed that cognitive reflection can predict the occurrence of behavioral anomalies, with these findings being similar to earlier studies. To measure cognitive reflection, we used the CRT (Frederick, 2005) to split subjects into two groups based on their CRT scores. We looked at the relationship between subjects’ CRT scores and risk and time preferences, finding that subjects with low CRT scores are risk averse in domain of gains. The results did not indicate any relationship between cognitive reflection and time preference. We also found that subjects with low CRT scores are more likely to exhibit the conjunction fallacy, illusion of control, overconfidence, base rate fallacy and conservatism. We found no relationship between cognitive reflection and either the self-serving bias or the status quo bias. Results additionally showed that the self-serving bias and illusion of control are exhibited more often among female subjects.

By replicating and extending previous results in Iran, the present study shows that they occur in other cultural contexts than where they were first found. Also noteworthy is that, in present-day Iran, the CRT is not so well known as it is in the English-speaking world, so most subjects were, we can assume, unfamiliar with it.

Cognitive reflection is one of many factors that can predict the incidence of behavioral anomalies. This study also used gender as a predictor, though it was related to only two anomalies that involved self-estimation. Study limitations include the fact that it was limited to university students, along with the fact that only seven biases were studied. Future studies could additionally consider the biases of financial investors.