Reflective liberals and intuitive conservatives: A look at the Cognitive Reflection Test and ideology

Abstract

Prior research finds that liberals and conservatives process information differently. Predispositions toward intuitive versus reflective thinking may help explain this individual level variation. There have been few direct tests of this hypothesis and the results from the handful of studies that do exist are contradictory. Here we report the results of a series of studies using the Cognitive Reflection Test (CRT) to investigate inclinations to be reflective and political orientation. We find a relationship between thinking style and political orientation and that these effects are particularly concentrated on social attitudes. We also find it harder to manipulate intuitive and reflective thinking than a number of prominent studies suggest. Priming manipulations used to induce reflection and intuition in published articles repeatedly fail in our studies. We conclude that conservatives—more specifically, social conservatives—tend to be dispositionally less reflective, social liberals tend to be dispositionally more reflective, and that the relationship between reflection and intuition and political attitudes may be more resistant to easy manipulation than existing research would suggest.

1 Introduction

Psychologists and political scientists have long argued that individual-level variation in ideology is driven, at least in part, by systematic differences in cognitive style and information processing (Eidelman et al. 2012; Pacini & Epstein 1999; Shook & Fazio 2009; Sidanius 1985; Tetlock 1983). Such differences appear to be dispositional and correlate with stable biological traits (Amodio et al., 2007; Kanai et al., 2011; Schreiber 2011). Several theoretical frameworks have been advanced to explain the connection between political worldview and cognitive or information processing styles. Notably, Jost, Glaser, Kruglanski & Sulloway (2003) synthesized a broad range of psychological frameworks into a theory of conservatism as motivated social cognition, in which conservatism stems from motivations to avoid threat and uncertainty. Others have attributed differences in cognitive style to differences in intelligence, with social conservatism being linked to lower IQ (Hodson & Busseri, 2012). While these frameworks are backed by empirical support, given their unflattering implications for political conservatives, they are unsurprisingly controversial.

An alternate possibility may be that ideological differences are associated with the extent to which a person relies on intuition versus being more reflective in making judgments. Intuition is characterized by quick, automatic and relatively effortless information processing, and reflection is characterized by slower, more deliberate and systematic reasoning (Stanovich, 2004). Although similar arguments are widely employed to explain political cognition (Marcus, 2012), little empirical research has been devoted to examining ideological differences using the same framework. The few studies that do exist are somewhat limited and come to contradictory conclusions (Iyer et al., 2012; Kahan 2013; Pennycook et al., 2012; see discussion below).

Significant support for expecting ideological differences associated with intuition and reflection can be traced to the success in explaining differences in individual-level religiosity (Pennycook et al. 2012, 2014; Royzman, Landy & Goodwin 2014; but see Piazza & Sousa, 2014). Several studies report that even fairly mild primes designed to induce intuition or reflection can affect religious attitudes, such that intuition is associated with higher reported levels of religious beliefs and vice versa for reflection (Gervais & Norenzayan, 2012; Shenhav et al. 2012). Given that ideology and religiosity both reflect coherent world views and that the empirical literature documents a strong correlation between political and religious attitudes in the U.S. (Brint & Abrutyn 2010; Hirsch et al. 2013; Layman & Carmines 1997; Malka et al. 2012), a natural corollary hypothesis is that American liberals and conservatives will differ in their dispositions toward reflection. These same studies on religious attitudes also raise the intriguing question of whether manipulating the use of reflection or intuition can effect political attitudes as well.

We seek to test this extension, whether intuition and reflection are systematic correlates of political attitudes and whether those same political attitudes can be manipulated by priming one or the other. Although we focus on ideological differences, given recent concerns about replicating results involving the manipulation of psychological constructs (Harris et al. 2013; Shanks et al. 2013), we assess the ability of previously documented tasks to influence reliance on intuition and reflection.

1.1 Reflection, intuition, and The Cognitive Reflection Test

Work in cognitive psychology has delineated between the use of intuition and reflection in attitudes and judgment making.¹ A great deal of variation and controversy surrounds the conceptual specifics and terminology of reflection and intuition (see Stanovich 1999, 2004 for reviews; see also Baron, Scott, Fincher & Metz 2014; Chaiken, et al. 1989; Keren & Shul 2009; Pennycook, Cheyne, Koehler & Fugelsang 2015; Sloman 1996). Specifically, there is some debate as to whether intuition and reflection operate sequentially—i.e. where intuition is automatic and then regulated through reflection (Evans, 2003, 2007)—or in parallel with individuals differing in the degree to which intuition versus reflection are activated in the first place (Baron et al., 2014; Sineyav & Peters 2015; Sloman 1996). However, it is largely accepted that tendencies to broadly rely more on reflection or intuition vary across individuals.

One tool for measuring individual differences in the use of reflective reasoning is the Cognitive Reflection Test (CRT; Frederick 2005), a three-item test designed with intuitive but incorrect responses and reflective, correct responses for each item. For example, one question asks, “A bat and a ball cost $1.10. The bat costs $1.00 more than the ball. How much does the ball cost?” The intuitive answer occurring to many people is $.10. However, through reflection, the correct answer of $.05 can be easily calculated. Thus higher scores on the CRT capture a tendency toward reflection. The CRT has been shown to be associated with a range of psychological traits, values, and beliefs (e.g., Barr, Pennycook, Stolz & Fugelslang 2015; Pennycook et al. 2014; Toplak et al. 2011).

Recent work has called into question the mechanisms underlying individual differences in CRT performance and its correlations with various psychological phenomena. Sinayev and Peters (2015) show that the predictive ability of CRT performance on various decision-making tasks is better explained by differences in numeracy rather than cognitive reflection. Baron et al. (2014) demonstrate that though the “sequential” view of dual process theory (i.e. initial intuitive responses are inhibited by higher-level reflection) may explain associations between CRT performance and certain tendencies, other associations with the CRT, including moral reasoning, lack evidence of a sequential process. Instead, parallel processing, whereby individuals differ in how they approach the CRT items in the first place seems more apt in these circumstances. It is beyond the scope of this study to determine which of these alternative mechanisms might explain a relationship between CRT performance and political attitudes, but covariation between political attitudes and reliance on reflection or intuition would nonetheless represent a critical step forward in understanding cognitive differences between liberals and conservatives.

1.2 Intuitive conservatives and reflective liberals

Several empirical studies point to the notion that liberals are more likely to be reflective than conservatives. For example, conservatives are more reliant on heuristics associated with implicit reasoning (e.g., Bizer et al. 2004; Jost et al. 2003; Rokeach 1948; Sargent 2004; Sidanius 1985; Tetlock 1983; Thorisdottir et al. 2007; Van Hiel, et al. 2004; Wilson et al. 1973), indicating less use of reflection and more reliance on intuition. Additional evidence comes from Eidelman and colleagues (2012), who find that increased blood alcohol levels—which tend to reinforce reliance on quick judgments and low effort thought—is associated with reporting more conservative political attitudes. More indirect evidence includes the relationship of intuition with conservative moral reasoning approaches and rationality with the moral reasoning of liberals (Garvey & Ford 2014), and the association between cultural thought styles that imply reflection or intuition with ideology (Talhelm et al., 2015). The empirical track record suggests a connection between lower levels of reflection and conservatism, overall.

A growing literature on the psychological underpinnings of political ideology provides a theoretical rationale for why a relationship between political attitudes and reflection versus intuition might exist. Some argue that American conservatism is inherently related to threat and uncertainty avoidance, which ultimately lead to an endorsement of inequality and resistance to social change (e.g., Jost et al. 2003). Conservatism is consistently associated with traits like need for cognitive closure and order, intolerance of ambiguity, and lower levels of cognitive complexity (Chirumbolo et al., 2004; Federico et al., 2005; Golec & Federico, 2004; Jost et al., 2007; Zavala et al., 2010). Conservatives tend to see the world in rigid categories, make judgments quickly without a willingness to seek more information or change their mind, and use simple stereotypes, even when it comes to interpreting and communicating policy statements (Tetlock, 1983). In essence, all of these seem likely to correspond with less reliance on reflection.

Social conservatism, in particular, may be negatively associated with reflection. Religion is highly interconnected with conservatism in the US context, especially following the 1980s (Brint & Abrutyn 2010; Hirsch et al., 2013; Layman & Carmines, 1997; Malka et al. 2012). In regards to social conservatism, Bouchard and colleagues outline the “Traditional Moral Values Triad” (TMVT), which explains how social conservatives organize their beliefs surrounding social institutions. The TMVT model argues that conservative beliefs are characterized by a strong obedience to authority that manifests iteself along three primary dimensions, including: 1) authoritarianism, or organization of the family; 2) religiousness, or control over the universe; and, 3) conservatism, societal organization (Bouchard, 2009; Koenig & Bouchard, 2006). The societal structure that social conservatives tend to prefer in all three circles involves strong leaders, rules, and hierarchy, which provide people with greater ease in discerning dominant people and institutions without much mental effort (Moors & De Houwer, 2005; Zitek & Tiedens, 2012). In addition, these organizational structures reduce the need of cognitive flexibility required to deal with any ambiguity in the world and allow for the use of quick judgments and simple stereotypes. Thus, the interconnection of beliefs surrounding the family, society and religion suggests that reflection should be related to social conservatism in the same way as it is connected to religious beliefs (Gervais & Norenzayan, 2012; Pennycook et al., 2012, 2014), as both provide a similar structure to different aspects of the world.

No more than a handful of published studies report co-variation between CRT scores and individual-level ideology. Iyer et al. (2012) and Pennycook et al. (2012) report negative relationships between conservatism and reflection, but in neither study was this relationship the main focus of the research nor was it explored or tested in-depth. Piazza and Sousa (2014) report no significant relationship between political conservatism and intuition as measured via the CRT, but again, this relationship was not the main focus of the study and not fully explored. Kahan (2013) was focused more centrally on this correlation between reflection and ideology, and found no significant relationship at all. As far as we are aware, these four studies represent the total published literature examining co-variation between measures of ideology and CRT scores and not only do they come to contradictory conclusions, but the results may not be directly comparable because of important sampling and measurement differences.

The first three studies used non-random samples. Iyer et al.’s (2012) sample was drawn from visitors to a web site and Pennycook et al. (2012) and Piazza and Sousa (2014) used samples from Amazon’s Mechanical Turk (these samples tend to skew liberal, see Berinsky, et al. 2012). Kahan’s (2013) study was based on a sample much better suited for making generalized conclusions—a representative, stratified sample of U.S. adults—but this study found no relationship.

The studies also use different measures of political ideology. Iyer et al. and Piazza and Sousa use single-item self-report measures of ideology (either 5-point or 7-point scales), while Kahan uses a combination of partisanship and ideological self-placement. The overarching problem with all of these measurement approaches to political attitudes boils down to a mixing of distinct constructs that are likely independently related to thinking style. Self-reported ideology is well-known to be a highly imperfect predictor of individual issue preferences (Converse, 1964; Ellis & Stimson, 2012; Jacoby, 1995; Goren, 2005; Greene, 2004; Green et al. 2004; Huddy, 2001; Stimson, 2004). Notably, in the United States, individuals who are socially and economically conservative, socially conservative but economically liberal, or socially liberal but economically conservative, all tend to self-identify as conservative. In contrast, self-identified liberals tend to be only those who are both socially and economically liberal (Ellis & Stimson, 2012). Other research corroborates the idea that social and economic preferences are distinct (Duckitt et al., 2002; Evans, Heath & Lalljee, 1996; Layman & Carsey, 2002; Stenner, 2005). Pennycook et al. did split conservatism into “fiscal” and “social” dimensions and showed that only the social dimension is significantly correlated with cognitive style, but these measures nonetheless still relied on subjects’ self-identification without measuring policy positions. Given that existing theories described above relate more to social and moral concerns, and prior findings show a connection between intuition and religiosity (Gervais & Norenzayan, 2012; Pennycook et al. 2012, 2014), we expect the relationship between inclinations towards a reliance on intuition or reflection and ideology to be driven more by social rather than economic issue preferences. We further expect specific issue stances to outperform broad self-reported ideology. None of the prior studies use ideology measures capable of teasing out this expected difference.

We seek to address all of these issues in the following studies. To increase external validity multiple samples (including one that is nationally representative) are used. To address the crucial issue of measurement we employ both a standard self-placement ideology scale and a modern version of the Wilson-Patterson (1968) index, a widely validated measure of ideology based on issue preferences. These procedures allow us to test whether use of reflection systematically varies with ideology generally and/or with particular categories of issue attitudes.

Additionally, we experimentally investigate whether political attitudes can be manipulated by priming intuition or reflection—something that no published study we are aware of has yet attempted (but see Swami et al., 2014, on manipulating conspiracist ideation). Motivation for this came from a series of studies where researchers used remarkably simple primes to induce reflection. For example, simply instructing subjects to think analytically (Rusou, et al., 2013), asking subjects to furrow their brows (Alter, et al., 2007), manipulating fonts to make words harder to read (Gervais & Norenzayan, 2012; Song & Schwarz, 2008), and using basic visual cues or particular patterns of words (Gervais & Norenzayan, 2012; Uhlmann et al., 2011) have all been suggested to induce reflection. Researchers using the latter priming strategies have successfully manipulated religious attitudes in published studies, so we employ these as a starting point, seeking to assess whether the impact on religious attitudes extends to political attitudes.

1.3 Study Overview

Our studies are designed to test the following hypotheses:

H1:

A disposition to be less reflective is positively associated with conservative political attitudes—particularly, with social conservative attitudes.

H2:

A disposition to be more reflective is positively associated with liberal political attitudes—particularly, with social liberal attitudes.

H3:

Subjects receiving an intuitive prime will report more conservative attitudes (especially for social issues) compared to those receiving a neutral prime.

H4:

Subjects receiving a reflective prime will report more liberal attitudes (especially for social issues) compared to those receiving a neutral prime.

In each of the studies we use the CRT as a measure of the tendency to be reflective (Frederick 2005). As previously stated, the connection between intuition and reflection—whether operating in parallel or sequential—has recently been debated. In accordance with this, although high scores on the CRT are likely indicative of reflection, low scores may not reliably indicate intuition (Pennycook et al. 2015). As such, we use less versus more reflection for H1 and H2, while using intuition versus reflection for H3 and H4 because the priming techniques involve intuitive and reflective manipulations, respectively. We employ a comprehensive measure of political attitudes, using both a 101-point self-placement ideology scale and variants of a Wilson-Patterson index. The latter allows us not only to capture a general, single dimension of ideology, but also to examine relationships between reflection and intuition with ideology across separate issue dimensions. This is critical because H1 and H2 posit the main impact of reflection will manifest itself in social attitudes; we expect the relationship between general political orientations and economic attitudes with reflection to be weaker. Tests of our first two hypotheses are based on straightforward correlations between CRT scores and political orientations. Tests of H3 and H4 are based on experiments using reflective and intuitive primes. Experiments 1–3 use sentence completion tasks to prime intuition or reflection and experiment 4 uses images to prime reflection.

2 Study 1

This study represents an initial investigation into how reflection and intuition relate to ideology and issue attitudes. Half of the subjects were used to examine individual differences in reflection and political attitudes (tests of H1 and H2), while half were used in a between-subjects, experimental design to examine whether priming intuition would lead to more conservative political preferences (test of H3). For the experimental group, the priming paradigm was adapted from Gervais and Norenzayan (2012; Studies 3–4), and involved having subjects create a four-word phrase from the five words given to them (this procedure is detailed below). We expected that the subjects given a set of words that included intuitive-related items would provide evidence of more conservative attitudes than those in the control condition—a straightforward replication of the one used by Gervais and Norenzayan (2012, Studies 3–4).

2.1 Subjects and design

We recruited two samples from Amazon’s Mechanical Turk (AMT) in the summer of 2012. Forty-seven of the 416 adult subjects were dropped due to attrition, leaving 369 subjects. Sample 1 completed the CRT and a set of survey items (n=190). Subjects in Sample 2 were randomly assigned to one of two conditions—control or intuition.² After removing 18 subjects from Sample 2 for typing the words exactly as they were given to them rather than making a four-word phrase, there were 81 subjects in the intuition condition and 78 in the control condition. Subjects were paid $0.50 for their participation in the experiment, which took less than 15 minutes. The questions were presented with Qualtrics.

Subjects in Sample 1 immediately received the CRT questions without receiving a priming task. CRT scores were created by adding the number of questions the subject answered correctly, ranging from zero to three. Following the three CRT questions, subjects were directed to a survey. The first question asked, “Labels are often misleading, but in general do you consider yourself liberal, conservative, or something in between?” Subjects reported their response using a 101-point slider, with one end labeled “very liberal” and the opposite, “very conservative” and were blind to the exact numerical location they were placing themselves on the slider. They were then asked to report their attitudes towards 19 different issues (including school prayer, gay marriage, stem cell research, biblical truth, abstinence only sex education, premarital sex, evolution, abortion, healthcare spending, welfare spending, government regulation of business, foreign aid, lowering taxes, illegal immigration, the death penalty, military spending, protecting gun rights, pacifism, and the torture of terrorist suspects). The list of issues was preceded by, “Here is a list of various topics. Please indicate how you feel about each topic.” Subjects reported their response to the prompt on a 7-point scale ranging from strongly disagree (1) to strongly agree (7) for each item. (Exact wording for the political attitude survey items for all four studies can be found in Appendix A.) We analyzed these issue positions in aggregate as well as divided into moral, punishment, and economic issues based on factor analyses described below. Both ideology and issue preferences were coded so that higher scores indicate more conservative positions in all four studies.

Subjects in Sample 2 were randomly assigned to one of two priming conditions. In the control condition, subjects were given 10 sets of five words. For each set, subjects were instructed to drop one word and create a meaningful phrase out of the remaining four. For example, if “wind”, “was”, “the”, “blowing”, and “retrace” were given, subjects could create the phrase, “the wind was blowing”. (Instructions and materials used for all four studies can be found in Appendix B.) All 10 sets of words for the control condition were taken from Gervais and Norenzayan (2012), who used these sets specifically because they do not include any words related to reflection or intuition. The intuition condition involved the same task, but five word sets included the target primes: impulse, hunch, gut, feels, and instinct. Immediately following the sentence completion task the subjects in both conditions completed the three CRT questions (to serve as a manipulation check) and then the same survey as the individual differences group.

2.2 Results

2.2.1 Individual differences (Sample 1)

The correlations between political attitudes and thinking style display a pattern consistent with the first two hypotheses, those with conservative issue preferences, especially towards social issues, are less likely to give correct answers to the CRT questions and liberals are more likely to do so. There is a significant negative correlation between CRT reflection and the conservative attitude index, consisting of all 19 items (α = .84; r = −.37, p < .001). For self-reported ideology, the coefficient is much smaller and is not significant (r = −.09, p = .21). The correlations between CRT scores and the individual political attitude items for all studies are provided in Appendix C.

In addition, we separated issue preferences into three categories to determine whether, as hypothesized, social domains of ideology were driving the overall results. A factor analysis using Varimax rotation was used to develop the sub-scales, which are available in Appendix C.) The moral issue scale (α = .88) consisted of attitudes toward school prayer, gay marriage, stem cell research, biblical truth, abstinence only sex education, premarital sex, evolution, and abortion. The economic scale (α = .80) consisted of healthcare spending, welfare spending, foreign aid, and government regulation of business. The punishment index (α =.73) consisted of attitudes regarding illegal immigration, the death penalty, military spending, and the torture of terrorist suspects. Protection of gun rights and lowering taxes did not load on any factor and pacifism loaded on the economic factor, which theoretically does not make sense, so these are not in the sub-indices. The sub-scales correlate with reflection in a similar pattern as overall issue attitudes. Moreover, the correlations between CRT scores and the narrower sets of issue scales consisting of moral (r = −.33, p < .001), punishment (r = −.22, p = .002), and economic (r = −.20, p = .006) attitudes are all statistically significant and in the expected direction.

To examine whether social attitudes held a stronger association with CRT scores than the sub-indices and overall, ideology, Steiger’s Z-tests were used to compare the correlations. Moral attitudes have as strong of a correlation with CRT as the punishment (Z = −.136, p = .17) and economic subsets (Z = −1.34, p = .18). In addition, punishment related issues were also equal in association as economic issues (Z = −.24, p = .81). However, the correlation with broad ideology and CRT was weaker than with moral (Z = −3.15, p = .002) and punishment attitudes (Z = −1.45, p = .15). So while social attitudes—moral and punishment items—did not differ in association with reflection, these were stronger than just looking at broad, self-reported ideology on a one-dimensional scale.

2.2.2 Experimental (Sample 2) results

Results of the ANOVAs for the experiment conducted with Sample 2, on the other hand, do not support the hypothesis that the manipulation produces differences across conditions (H3). A ready explanation may exist for this null result. In contrast to previous research, the manipulation check shows those assigned to the intuition condition in our experiment did not differ on CRT scores (F(1, 157) = .06, p = .800). In other words, the priming protocol failed to influence intuition. Given this, unsurprisingly there were no mean differences in self-reported ideology (F(1, 157) = 1.77, p = .185) or the conservative issue index (F(1, 157) = .46, p = .500). Looking at the different sub-indices, there again were no mean differences between the two groups on social, punishment, or economic policies with all of the F values being less than one.

2.3 Discussion

There are two main findings from the first study. First, we found support for H1 and H2 in that conservative attitudes and CRT scores are correlated. While the relationship for social conservatism (i.e., moral and punishment issues) was stronger than for broader ideology, it was equal to economic issue preferences. Second, we found no support for H3: the intuitive primes did not seem to shift individuals into a less reflective state and, given this, unsurprisingly there were no significant differences in political attitudes between our experimental and control conditions.

3 Study 2

Even though the experimental portion of Study 1 used a priming strategy based in published research, it failed to manipulate intuition and this clearly explains why we failed to find ideological differences between the experimental and control conditions. There are several possibilities for why the manipulation failed. One is that the prime, despite being aimed at triggering intuition, nonetheless involved problem solving and so actually triggered reflection. However, the same prime has been used before to prime intuition (Uhlmann et al. 2011). Perhaps a more likely reason for the failure of the prime is that it may simply be easier to activate reflection rather than intuition. Certainly the bulk of relevant experimental literature uses reflective rather than intuitive primes for manipulations.

Study 2 seeks to examine both of these possibilities. Here we replicate the basic blueprint of the experiment from Study 1 but use a more representative sample of the United States and add a third, reflective condition to the experimental design. Subjects in the control condition are used as a platform to re-test the individual difference findings from Study 1 to determine if the findings hold up in a more representative sample.

3.1 Subjects and design

Through a Time Sharing Experiments for the Social Sciences (TESS) grant, 870 respondents were recruited through GfK (formerly Knowledge Networks) of which 536 completed the study. GfK recruits panels of subjects through address-based sampling and random digit dialing, employing national random samples that are representative of the United States population with post-stratification weights based on the Current Population Survey to reduce non-response bias. Eighteen subjects were removed because they failed to type in a four-word phrase for the priming task. We used a between-group experimental design with subjects randomly assigned to an intuition (n=172), control (n=173), or reflection (n=170) condition.

Potential subjects were emailed by GfK informing them of an available study and given a link to complete the experimental protocol. After random assignment to a condition, subjects in the reflection and intuition conditions completed the same sentence completion task described in Study 1. For the control and reflective conditions, all ten word sets came from Gervais and Norenzayan (2012, Studies 3 and 4). Half of the word sets for the reflective condition contained the reflection related words of reason, ponder, think, rational, and analyze. The five target manipulation words for the intuitive condition, hunch, feels, instinct, intuition, and emotions, were taken from Uhlmann et al. (2011) who successfully induced intuition using a sentence completion prime in their study on moral attitudes.

Once the priming task was completed, subjects answered a short survey that included the same ideological self-identification question from Study 1. Additionally, using 5-point response scales, subjects were asked about their attitude towards the legality of abortion in cases that a woman’s health could be compromised, same-sex marriage, prayer in public school, spending on welfare programs, price regulation by the government, government-run health insurance, defense spending, the death penalty, and torture of terrorist suspects (α = .75; see Appendix A for the exact wording). We sub-divided the issues into the same three categories described in Study 1. The moral policy sub-index includes abortion, same-sex marriage, and school prayer (α = .66). Punishment issues include the death penalty, spending on military defense, and torture (.69). Economic issues include welfare spending, government regulation, and health insurance (.63).³ The final task had all subjects complete the CRT.

3.2 Results

In order to test the robustness of the individual differences reported from Study 1, we first use the control condition in Study 2 to run correlational analyses between CRT scores and political attitudes, replicating the Study 1 results for H1/H2 using a more representative sample. Again, we find support for these hypotheses. We find moderately sized negative correlations between CRT scores and conservative issue preferences overall (r = −.23, p = .003). This general relationship holds for moral (r = −.22, p = .003) and punishment-related policies (r = −.34, p < .001), but disappears entirely with economic policy issues (r = .07, p = .36). The correlation for self-reported ideology is also in the predicted direction and significant (r = −.19, p = .02). Results of Steiger’s Z tests show that the relationship between CRT and economic issues is weaker than for moral (Z = −3.60, p < .001) and punishment (Z = 2.98, p = .003) attitudes. In addition, while the punishment issue relationship (Z = −2.08, p = .038) is significantly stronger than self-reported ideology, the moral one was not (Z = −.53, p = .600).

The experimental results for Study 2 parallel the results from Study 1. Between-group ANOVA results show no differences in the number of correct CRT answers provided by the subjects between any group (F(2, 512) = .35, p = .707) indicating that the priming protocol manipulation again failed to induce any higher or lower amounts of reflection. With no differences in thinking style, again unsurprisingly, there were no statistically significant differences between the groups on self-reported ideology (F(2, 475) = 1.17, p = .305) or issue attitudes (F(2, 512) = .57, p = .564). Looking at the sets of specific issues, again there are no significant differences among the groups in mean attitudes towards moral, punishment, and economic attitudes.

3.3 Discussion

In Study 2 we again find correlational evidence to support the hypothesis that higher CRT scores are associated with being liberal. Again, there is evidence that the expected relationship between reflection and political orientations is centered more on social issues than economic issues and, in the case of punishment, self-reported ideology more generally. One reason these results are not stronger may be due to the fact that resource limitations dictated a shortened Wilson Patterson battery, i.e. we had fewer issue items to construct our scales from and thus may have had weaker measures in our analysis.

The Study 2 experiment again fails to find any evidence that primes used in previous research do anything to actually induce reflection or intuition. Unsurprisingly, a non-existent priming effect did not lead to any detectable shifts in political attitudes. While we originally attributed this to the difficulty in priming intuition, it may be that priming reflection is difficult as well. Recent research has indicated replication of widely accepted primes often fails (Meyer, et al. 2015; Thompson, et al. 2013).

4 Study 3

Study 3 is a straightforward replication of Study 2 but with AMT subjects. We wanted to see how robust our results for H1/H2 were when we used an expanded set of issue items and, for the third time, try to get established priming protocols to demonstrate the impact on intuition and reflection reported in previous research that is required for tests of H3 and H4.

4.1 Subjects and design

We recruited 773 subjects from AMT and each was credited $0.50 for participating. Thirty-nine were dropped because they failed to complete the primes or dependent variables and/or the CRT task. The between group experiment was the same as described in Study 2 with subjects randomly assigned to either the control (n=255), intuitive (n=243), or reflective (n=236) condition. Once subjects finished the priming task, they filled out a short survey almost identical to that described for the previous studies above. Twenty-four individuals were dropped due to not completing the priming task by not making a coherent four-word phrase or typing in random words. This leaves 249 in the control, 237 in the intuitive, and 224 in the reflective.

4.2 Procedure

AMT workers were able to self-select into the study after reading a short description of the project on the platform. Once they chose to participate, they were moved to the main experiment in Qualtrics. Following the same manipulation task described in Study 2, subjects were asked to report their ideology and to indicate their attitudes toward 20 issue positions using the same Wilson-Patterson prompt and response options as in Study 1. Most of the issues are the same in this study except that foreign aid was removed and agreement with small government and global warming caused by humans were added. Following the political attitude items, all subjects completed the three CRT items and reported demographic information.

4.3 Results

We again used the control condition to test for individual differences in engagement in each thinking style and political attitudes. These results replicate the key findings taken from Study 1 and Study 2—CRT scores reliably correlate with political attitudes.

Overall, CRT scores were correlated with overall issue attitudes as measured by the full Wilson-Patterson Index (α = .91; r = −.21, p = .001). As in previous studies, we divided the Wilson Patterson battery into three subscales. The moral issue subscale includes prayer in school, premarital sex, gay marriage, abortion, evolution, biblical truth, stem cell research, and abstinence only education (α = .90). The economic subscale includes healthcare spending, welfare spending, business regulation, small government, and lower taxes (α = .86). Finally, the punishment scale included illegal immigration, the death penalty, military spending, and torture of terrorist suspects (α = .74).⁴ These subscales continue the trend showing that CRT scores are associated with attitudes on moral issues (r = −.18, p = .005) and punishment-related policies (r = −.24, p < .001) but not for economic policies (r = −.09, p = .147). The correlation between CRT scores and self-reported ideology was also significant (r = −.18, p = .005). Also, while the correlation between CRT and economic issues equals that of the moral correlation (Z = 1.22, p = .222), the punishment correlation is stronger (Z = 2.55, p = .011). The moral and punishment correlations are equal as well (Z = .88, p = .378). Last, with ideology demonstrating a significant negative relationship with CRT, the correlation is equal to the social (Z = .02, p = .981), and punishment correlations (Z = 1.09, p = .275).

The experimental results from Study 3, like the correlational results, replicated the findings from previous studies. Again we found no evidence that the priming protocols established in previously published studies pushed people to be more or less reflective (F(2, 709) = .34, p = .714). As in Studies 1 and 2, the manipulation also did not lead to differences in self-reported ideology (F(2, 707) = 1.32, p = .267) or issue preferences (F(2, 709) = 1.33, p = .264). Breaking the issue preferences down into the three groups—moral, punishment, and economic—again did not yield any significant differences between the groups.

4.4 Discussion

In the third study we again are able to show a consistent pattern of individual differences in level of reflection and political attitudes. Those who display increased reflection are more likely to self-report being liberal and have liberal policy attitudes. However, only one of the two indices consisting of social attitudes, punishment related policies, have a stronger relationship with the CRT than the economic correlations. In short, we again find consistent evidence supporting H1 and H2.

In the experimental portion of Study 3 we again attempted to manipulate reflective and intuitive thinking using the sentence completion task employed by Gervais and Norenzayan (2012) and Uhlmann et al. (2012), expecting that this prime would shift political attitudes to the left or right. Again, the prime did not alter performance on the CRT and thus, unsurprisingly, did not alter political attitudes. The consistent results from experiments on three distinct samples leaves two obvious possibilities to explain our results: (1) The sentence fluency task is not priming the target behaviors, or (2) Priming individuals to engage reflection or intuition is not as readily successful as suggested by Gervais and Norenzayan (2012) and Uhlmann et al. (2012) (also see Meyer et al. 2015 and Thompson et al. 2013). Both of these are a possibility given that replications in simple priming tasks have failed to replicate and, in conjunction, they may be too unstable to work for an extended survey. In addition, the sentence fluency task involves creative problem solving to some degree, which may be cancelling out any effect of the intuitive-prime. In the next study we use a different priming strategy, a more passive one, to determine if the null results are partly due to the specific task.

5 Study 4

So far we have found clear evidence linking reflection and political attitudes. Reflection and intuition, however, seems to be “sticky”. In three attempts we have yet to successfully manipulate it, let alone find evidence that such manipulations push around political attitudes. In Study 4 we switch priming strategies, using the image primes employed in Experiment 2 of Gervais and Norenzayan (2012) to try to prime reflection. Specifically we prime reflection using images of The Thinker by Rodin. The control group, which was given alternative images, is again used to examine individual differences between reflection and policy preferences and ideology.

5.1 Subjects and design

We recruited undergraduate students from introductory Political Science courses during the Fall 2013 and Spring 2014 semester. A total of 281 participated in the experiment (71 from the spring) and thirty-four were dropped for not completing the protocol. There were no differences between the two semesters on condition assignment based on chi-square tests and ANOVAs showed no mean differences between subjects from different semesters on any of the main independent or dependent variables. Therefore we combine both samples to perform the analysis.

The experiment was set up as a between-group design where subjects were randomly assigned to one of the two conditions—reflective (n=124) or control (n=123)—with both conditions getting visual primes very similar to those used in Gervais and Norenzayan (2012, Study 2).

Students were contacted by email and asked to participate in exchange for one research credit (the course required completing eight research credits). Students who chose to participate were directed to take the link to the experiment, programmed in Qualtrics. After giving consent, subjects were randomly assigned to the reflective condition and shown four distinct pictures of Rodin’s The Thinker (which depicts a reflective pose) or the control condition and shown four distinct images of the statue, Discobolus of Myron. Each image was displayed for 30 seconds before subjects were allowed to continue. Examples are in the Appendix.

Following the primes, subjects completed the same survey used in Study 3. Subjects were asked their agreement or disagreement on 20 issue positions (α = .89), and to self-report their ideology. The only difference was the global warming item was replaced by foreign aid. As was done previously, the issue items were broken down into a moral index consisting of the same eight moral issues (α = .89), a punishment index consisting of gun rights, immigration, the death penalty, and torture or terrorism suspects (α = .75), and an economic index using the healthcare, welfare, business regulation, and small government items (α = .82).⁵ At the end subjects completed the CRT and were asked two additional questions—to name the statue they viewed and to state why they thought they were asked to look at it. Five subjects (four in the reflective condition) reported that the statues were used to increase their level of thinking. Removal of these five had no impact on our results so we kept them in the final analysis.

5.2 Results

In order to test individual differences among political attitudes and reflection we used subjects in the control group (H1 and H2). The results here are slightly weaker than in the previous samples (perhaps because of the limitations inherent in a student sample), but the general pattern is clear enough. That pattern replicates the findings of the previous three studies: higher CRT scores are associated with more liberal policy attitudes (r = −.16, p = .088). This relationship is stronger for moral (r = −.23, p = .010) compared to economic (r = .04, p = .670; Z = 2.05, p = .040) and punishment issues (r = −.01, p = .950; Z = 1.94, p = .052). Self-reported ideology (r = −.11, p = .242) is in the expected direction but not significant. However, the size of the ideology correlation is not different from the social (Z = 1.58, p = .113) or punishment policy correlations (Z = 1.14, p = .253).

While the evidence supporting H1 and H2 is slightly weaker for this sample, our experimental results also repeat the priming failures reported from the first three studies. Mean differences on CRT scores for the subjects in the reflective condition did not differ from those in the control condition (F(1, 245) = .12, p = .727). With no detectable priming effect, unsurprisingly there were no significant mean differences on any of political attitude scales.

5.3 Discussion

While the results of Study 4 are not as strong as the previous ones, it confirms the pattern of key findings of the previous studies. The correlational patterns between reflection and political attitudes are statistically weaker but they maintain the expected pattern of higher levels of reflection being associated with more liberal attitudes. Also, despite using an alternate priming style with a reported large effect in previous studies (see Gervais & Norenzayan, 2012, Study 2), we again saw no significant between-group differences. The general inference continues to be that liberals and conservatives, and social conservatives more so, use different thinking styles, and those cognitive patterns are resistant to manipulation.

6 General discussion

In four separate studies we find a consistent pattern showing that those more likely to engage in reflection are more likely to have liberal political attitudes while those less likely to do so are more likely to have conservative attitudes. These findings offer consistent support for H1 and H2. All four Wilson Patterson indices using all issue attitudes are negative and three are statistically significant at p < .05 (the fourth is significant at p < .10). Notably, we find some evidence that individual differences in reflection are tied to socially conservative issue preferences more than economic issue preferences or broader ideological self-identification.

In contrast to the policy attitudes, the one-dimension self-reported ideology measure does not appear to be strongly associated with reflection. CRT scores are only significantly negatively related to self-reported ideology in two of the samples. The lack of correlation between CRT scores and self-reported ideology, overall, suggest that this measure, used in much of the previous literature looking into the connection between reflection and ideology is misleading.

While finding consistent evidence to support H1 and H2, we found no evidence at all to support H3 and H4. More accurately we found no evidence that priming strategies used in previous studies have any impact on reflection, and thus could not really test H3 and H4. We were repeatedly unsuccessful in priming intuition (Studies 1–3) or reflection (Studies 2–4), and given the lack of a priming effect we unsurprisingly found no differences in political attitudes across our experimental conditions.

There are several possible explanations for why we did not replicate previous research. One reason might be that the CRT itself acts as a prime that induces reflection. The CRT is not an easy task, and generally the mean number of correct answers (out of 3) on the CRT in web-based and college samples is between .5 and 1 (Frederick 2005; Pennycook et al. 2015). It is thus possible that simply embarking on the CRT pushes people toward reflection and so any effects of the primes were cancelled out by completion of the CRT. However, this study (and many others) nonetheless finds variation in CRT scores that systematically correlates with other traits, and so it does not seem to be the case that CRT pushes everyone toward reflection (at least not enough to wipe out variation in the variable). Another possibility is that the battery of survey questions asked between the primes and CRT washed out the effects of the prime. However, if this was the case, it indicates that, even if there were slight changes in information processing after the primes, the effects were extremely small and of little substantive value. Finally, it is possible that our measures of political attitudes tapped relatively well-formed opinions and beliefs and effects might actually be observed if the measure involved a more novel decision-making task regarding political issues, but this would not explain the fact that CRT scores were not influenced in the first place.

We thus conclude that, although reflection can help explain individual level differences in political attitudes, individual tendencies toward intuition or reflection are much more stable and resistant to easy manipulation than reported by previous studies. To be blunt, in four attempts we found no evidence at all that previously used reflective and intuitive priming strategies worked. The inability of standard experimental manipulations to shift levels of reflection and intuition suggest that conservatives’ tendency to be less reflective than liberals is more of a dispositional trait difference rather than an environmentally-mediated state. At an absolute minimum, if these tendencies can be influenced in a way that leads to differences in political attitudes, our findings suggest it will require significantly different strategies than the some of the standard priming protocols employed in other studies.

What might explain social liberals’ tendency toward reflection more than a social conservative’s? We suggest that the relationship emanates from the psychological traits that tend to covary with political ideology, as discussed earlier. Specifically, much work has shown that political conservatism is associated with avoidance of uncertainty and preferences for hierarchy, rules, and structure (Federico, Golec & Dial, 2005; Golec & Federico, 2004; Jost et al. 2003; Jost et al., 2007). A socially conservative society offers such structure. In a socially conservative society less cognitive effort is required to make decisions and identify dominant individuals and institutions (Moors & De Houwer, 2005; Zitek & Tiedens, 2012), and so intuitive thinking lends itself naturally to the endorsement of such. Reflection, on the other hand, acknowledges and even embraces ambiguity and conflict, and it is in many ways hindered by rules that impose structure and hierarchy, and so it makes sense that individuals with a greater tendency toward reflection would prefer socially liberal policies.

Thus, in our view, it should not be concluded that the results presented herein suggest social conservatism is generally wrong because the people who endorse it tend to be less reflective, or that social conservatism is unattractive to people who are prone to reflection because it is logically indefensible. More likely, we suggest, is that social conservatism (similarly to religious beliefs and as described above) grows out of a high valuation of structure and cognitive consistency, which is naturally complemented by intuitive thinking, whereas social liberalism grows out of the high tolerance of uncertainty and valuing of cognitive complexity that is complemented by reflection. Indeed, the very political issues that we use to look at social conservatism are those in which guidance from moral authorities (whether they be religious or political) give concrete answers—i.e. abortion is always wrong, or illegal immigration is definitely a threat to the United States. Alternatively, conclusions drawn on these topics after reflective thinking may yield ambiguity—what about abortion for rape or incest victims? What about illegal immigrants who come here as a result of poverty or persecution?

6.1 Conclusion

The results of the studies reported above offer clear and consistent support to the idea that liberals are more likely to be reflective compared to conservatives. In many cases, sets of social issues rather than economic issues were a stronger force in the relationship. Specifically, those with a lower propensity to use reflection were more likely to hold conservative policy attitudes relating to morality and punishment. Economic attitudes and self-reported ideology did not display such a consistent pattern in relation to reflection or intuition. We were able to replicate this general finding across four different samples, using responses to the CRT items. However, we were unable to show, using previously successful priming techniques, that this relationship is subject to manipulation.