The unconscious thought advantage: Further replication failures from a search for confirmatory evidence

Abstract

According to the deliberation without attention (DWA) hypothesis, people facing a difficult choice will make a better decision after a period of distraction than after an equally long period of conscious deliberation, an effect referred to as the unconscious thought advantage (UTA). The status of the DWA hypothesis is controversial, as many studies have tried but failed to replicate the UTA. Here, we report a series of experiments that sought to identify the conditions under which the UTA can be replicated. Our starting point was a recent meta-analysis that identified the conditions under which the UTA was strongest in previous studies. Using a within-subjects design and a task that met each of these conditions, we failed to replicate the UTA. Based on closer inspection of previous methods and findings, we then examined some additional factors that could be important for replicating the UTA, including mental fatigue and choice complexity. This was to no avail, as the results revealed only a significant conscious thought advantage, when choice complexity was increased relative to the first experiment. We subsequently conducted exploratory analyses on the data across experiments and found that male subjects showed a significant conscious thought advantage while female subjects showed a trend towards an UTA. Taken together, our results suggest that replication of the UTA may depend more on characteristics of the sample than on the characteristics of the task, and they suggest that gender could be a source of variance in the outcomes of previous studies using a between-subjects design.

1 Introduction

Throughout life, people face a variety of decisions that involve choices between options with many relevant attributes. For instance, in deciding which university program to enroll in, one needs to consider such goals as getting the best possible education, minimizing distance to family and friends, and finding a nice place to live. A rational approach to solving such a complex decision problem would involve comparing the different options by evaluating the extent to which they satisfy these goals (e.g., Baron, 2008). To perform such a decision analysis, one would preferably use external aids such as a computer or a pencil and paper for writing out the decision problem and performing the required computations. But what if you do not have the option of using external aids, and instead have to rely on memory and mental information processing capacity for making complex decisions?

While conventional wisdom suggests that one should think carefully before making a difficult choice, the deliberation without attention (DWA) hypothesis proposed by Dijksterhuis and colleagues (Dijksterhuis, 2004; Dijksterhuis, Bos, Nordgren, & Van Baaren, 2006; Dijksterhuis & Nordgren, 2006) argues otherwise. Specifically this hypothesis claims that complex decisions are best made without conscious deliberation, that is, following a period during which conscious deliberation about the choice at hand is prevented by means of performing an unrelated attention-demanding task. In support of this hypothesis, Dijksterhuis et al. (2006) reported the results of an experiment that consisted of three phases. In the information acquisition phase, subjects received information about four fictional cars, which were each described in terms of 12 characteristics that could be positive (“is fuel efficient”) or negative (“does not have airbags”). One of the cars had nine positive characteristics whereas two had six and one had only three. The characteristics were presented one after the other and the subjects were instructed to form a thorough impression of each car. Following this information acquisition phase, one group of subjects was told they had four minutes to think carefully about the options before they would be asked to make a choice. This comprised the “conscious thought” condition. The other group was told they would first have to do another task—solving anagrams—for four minutes before they would be asked to make a choice. This comprised the “unconscious thought” condition. The results of this between-subjects comparison showed that the subjects in the unconscious thought condition performed better than those in the conscious deliberation condition, with the latter performing at chance in picking out the best car. In other words, subjects who were given the opportunity to think about the cars made worse choices than those who did not think about the cars due to the distracting task.

To explain their surprising finding, Dijksterhuis and Nordgren (2006) proposed the aforementioned DWA hypothesis. This hypothesis claims that the processing of the earlier presented information will continue at an unconscious level during the execution of the distracting task, effectively producing a preference that will guide one’s decision when a choice eventually needs to be made. To explain why this unconscious form of deliberation produces better choices than conscious deliberation, Dijksterhuis and colleagues further proposed that unconscious thought has a much greater capacity for processing information than conscious thought. Hence, unconscious thought would be better suited for dealing with complex decisions and, as a consequence, momentary distraction would result in an unconscious thought advantage (UTA) in case of decisions involving many options with many characteristics of relevance.

In claiming that people should rely on unconscious processing to make better choices, the DWA hypothesis readily attracted attention from the public media, and it readily attracted criticism from other scientists. Dijksterhuis’ claims were considered misleading and potentially harmful (Bleker, 2006), and researchers were quick to offer alternative explanations for the UTA (Shanks, 2006; see also, Gonzalez-Vallejo, Lassiter, Belleza, & Lindberg, 2008). Nevertheless, the unconscious thought paradigm took flight as researchers started to investigate the boundary conditions of the UTA. This work indicated that the UTA is not a very robust phenomenon as many studies failed to replicate it, in spite of using similar methods to those that Dijksterhuis used (e.g., Acker, 2008; Calvillo & Penaloza, 2009; Lassiter, Lindberg, Gonzalez-Vallejo, Belleza, & Phillips, 2009; Newell, Wong, Cheung, & Rakow, 2009; Rey, Goldstein, & Perruchet, 2009; Thorsteinson & Withrow, 2009; Waroquier, Marchiori, Klein, & Cleeremans, 2009). At the same time, however, Dijksterhuis and other researchers produced a number of replications and extensions of the UTA (e.g., Bos, Dijksterhuis, & Van Baaren, 2008; Ham, Van den Bos, & Van Doorn, 2009; Lerouge, 2009; Smith, Dijksterhuis, & Wigboldus, 2008). In short, the available literature currently features 30 published studies that together report a total of 55 experiments in which the quality of judgments and decisions was compared between a conscious and an unconscious deliberation condition (see Table 1). Of these experiments, 25 replicated the UTA in some form, while 26 showed no significant difference in performance, and four showed significantly better performance in the conscious thought condition, that is, a conscious thought advantage (CTA).

Table 1: Overview of published studies contrasting the quality of choices made based on conscious and unconscious thought. Studies examining product satisfaction are not included as the extent of product satisfaction can not be considered to be equivalent to choosing the best of several options. CTA = conscious thought advantage, UTA = unconscious thought advantage, ns = non-significant difference.

¹ Dijksterhuis et al. (2009) claim that soccer experts show a significant unconscious thought advantage in predicting the outcomes of soccer matches. Gonzalez-Vallejo and Phillips (2010) re-analyzed the data and showed that this conclusion was not justified.

² For this experiment, Dijksterhuis (2004) did not report a statistical test for the comparison of performance in the conscious and unconscious deliberation conditions. However, we can deduce that this comparison would have yielded a non-significant difference because Dijksterhuis did report a statistical test for the comparison of the unconscious deliberation condition and a condition in which subjects had to make a choice directly after the information acquisition phase. This comparison was reported to yield a significant difference on the basis of a Chi-square test with “χ² (59, N = 60) = 3.13, p < .04, one-tailed”. However, this constitutes a type 1 error because the critical value of the χ² statistic is 3.84 for a one-tailed test. Accordingly, we conclude that the test of the even smaller difference between the unconscious and conscious judgment conditions could only yield a non-significant difference.

³ Mamede et al. (2010) found that for complex cases, physicians made significantly better diagnoses in a conscious deliberation condition than in an unconscious deliberation condition. For studentsin medicine, there was no difference in performance between these conditions.

⁴ Newell et al. (2009) found that subjects in a conscious thought condition were better able to differentiate amongst the four choice options than subjects in an unconscious deliberation condition.

⁵ Waroquier et al. (2009) found that subjects’ ratings of four options better matched the objective quality of the options in a conscious thought condition.

Given that most of the experiments listed in Table 1 used methods and materials similar to those used by Dijksterhuis (Dijksterhuis et al., 2006), an intriguing question is why the results of these experiments could be so inconsistent. To address this matter, Strick and Dijksterhuis and colleagues conducted a meta-analysis on a comprehensive collection of published and unpublished datasets, and they examined a number of methodological details that differed between studies (Strick et al., 2011). The results showed that averaged across studies, the UTA remained significant with a pooled effect size of.219. More importantly, the results also showed that there are indeed quite a few methodological details that have a significant influence the magnitude of the UTA.¹ These moderators not only pertained to the nature of the choices that had to be made, but also to manner in which the information acquisition and deliberation phases were implemented in different studies. Specifically, the analysis by Strick et al. showed that effect sizes were larger in studies that employed more complex decisions, as predicted by the DWA hypothesis. The analysis also showed that the UTA is absent when subjects have a different goal than evaluating the options for a later judgment or choice (as in most studies), and when subjects are led to focus on specific aspects of the information instead of forming a general impression of each of the options (as in most studies). In addition, the UTA was found to be stronger when the information about the choice alternatives was presented grouped per alternative instead of randomly intermixed across alternatives, when this information was presented at a relatively high pace, and when this information was accompanied by pictorial information. With regard to the deliberation phase, the analysis showed that the UTA was stronger when the duration of this phase was relatively short, and when the task used to distract subjects in the unconscious thought condition involved solving word-search puzzles instead of solving anagrams, or performing a verbal working memory task.

1.1 Outline of the current study

In the current study, we set out to examine the conditions under which the UTA can be reliably obtained. In accordance with Strick et al. (2011), we assumed that the effect is real but specific to certain conditions that were not met in the studies that failed to replicate the UTA. The starting point in our search for confirmatory evidence was an experiment in which we examined whether the UTA would be obtained in a task that met each of the conditions identified by Strick et al. (2011). Since this experiment did not yield a significant UTA, we then further scrutinized the methods and results of previous studies to examine what other factors could be involved in replicating the UTA. This analysis revealed one factor that was not examined in the meta-analysis by Strick et al.: the level of performance in the conscious thought condition. Specifically, we found that performance in the conscious thought condition was at chance in all but one of the studies that did replicate the UTA. In striking contrast, only one of the studies that showed above-chance performance in this condition showed a significant UTA. Thus, our search for the UTA continued by examining which factors might lead to lower performance in the conscious thought condition, thereby increasing the probability for observing a significant UTA. This resulted in three further experiments in which we examined whether the UTA might occur under conditions of mental or decision fatigue, and whether it might occur when the complexity of choices is further increased by increasing the number of options and attributes involved in a choice.

Auxiliary issues addressed in the experiments. Across the four experiments, we also manipulated and assessed a number of other aspects that we considered to be of relevance to the observation and interpretation of a significant UTA. To start, we chose to use a within-subjects design wherein each subject made two choices, once after deliberation and once after distraction, with the order of the two types of choices counterbalanced between subjects. The main reason for using this design instead of the between-subjects design used in nearly all previous UTA studies (the sole exception can be found in Mamede et al., 2010) was to preclude the possibility that any difference between performance could stem from a spurious between-group difference. In view of the fact that the requirement to make two choices instead of just one could lead a within-subjects design to produce different results than a between-subjects design (e.g., Greenwald, 1976), we also conducted between-subject comparisons to determine if performance in the conscious and unconscious thought conditions differed depending on whether these conditions were done first or second.

Aside from attempting to constrain the variance that might arise from spurious between-group differences in a between-subjects design, we also attempted to control for a number of other possible sources of variance. The first regards variance that might arise from the use of a relatively long and fixed conscious deliberation phase. While programming our first experiment we noticed that a deliberation phase of three or four minutes is rather long and then discovered that there is no solid empirical basis for why this phase should take this long. To wit, the sole motivation for why the deliberation phase should be so long can be found in Dijksterhuis et al. (2006) who stated that “The choice to choose a conscious thought (and unconscious thought) period of four minutes was based on earlier testing where subjects were given different amounts of time to think and were asked whether the amount of time given was satisfactory. In experiments such as this, most people indicate that three to four minutes is enough” (Dijksterhuis et al., 2006; p. 3, in supplementary method section). Importantly, however, the only published dataset pertaining to this issue suggests that subjects may need only 33-49 seconds to arrive at a decision (Payne, Samper, Bettman, & Luce, 2008), thus suggesting that a deliberation phase of three or more minutes is unnecessarily long.

To gain further insight into this matter, we asked subjects after they had completed the choice task how much time they thought they had needed to reach a decision in the conscious deliberation condition. In addition, we asked them what they did in the remaining period of the deliberation phase. Assuming that conscious thought leads to poor decisions, while being distracted—for instance by thinking about issues unrelated to the choice problem—would lead to more accurate decisions through promoting unconscious thought, we considered it important to know how much time subjects need to make their choice in the conscious deliberation phase.

In addition to assessing how much time subjects truly spent on conscious, and, possibly, inadvertent unconscious deliberation during the conscious deliberation phase, we also examined whether subjects engage in inadvertent conscious deliberation when they need to make their choice after performing the distracting task in the unconscious deliberation condition. Since making this choice would require the retrieval and evaluation of the information presented prior to executing the distracting task, it stands to reason that subjects might inadvertently engage in conscious thought after completing the distracting task. If so, then this could lead to worse performance, reducing the probability of observing the UTA. To preclude this possibility, we incorporated a speeded response requirement in the unconscious deliberation condition and we assessed how long subjects needed to make a choice after completing the distracting task. Since the use of such a speeded response requirement has not been documented in previous studies examining the UTA² we also included a condition without a speeded response requirement so as to enable a comparison of decision times and choice accuracy across these two versions of the task.

Lastly, we also examined the relationship between performance in the choice task and performance on the cognitive reflection task (CRT; Frederick, 2005). This is a three-item test that assesses the extent to which someone relies on intuition or analytical thinking in solving problems. In particular, each item on the test concerns a mathematical problem that automatically brings to mind an answer that is incorrect. For instance, one of the items consists of the following statement: “A bat and a ball cost $1.10 together. The bat costs $1 more than the ball. How much does one ball cost?” Here, the answer that comes to mind spontaneously is 10 cents, but this answer is incorrect. To find out that this answer is incorrect and to eventually arrive at the correct answer of 5 cents, one needs to apply analytical thinking to arrive at the appropriate mathematical formulation to solve the problem. Thus, people scoring high on this test can be said to have strong analytical thinking skills whereas those scoring low can be said to rely more on their intuitions. By including this test we could examine if and how performance in the conscious and unconscious deliberation conditions is related to reliance on intuitive vs. analytical thinking.

2 Experiment 1

The primary aim of Experiment 1 was to ask whether the UTA would be replicated in an experiment that met each of the conditions under which this effect was found to be strongest in previous studies according to the meta-analysis by Strick et al. (2011; see Table 2). In addition, we assessed how long subjects thought they needed to think through their choice in the conscious thought condition, and we manipulated the nature of the choice instruction in the unconscious thought condition, with half of all subjects receiving an instruction that emphasized the need to respond rapidly and without further thought while the other half were simply asked to make a choice, thus allowing them to respond at their own leisure.

Table 2: Moderators of the UTA (Strick et al., 2011), and the manner in which these conditions were incorporated in Experiment 1.

2.1 Method

Subjects. Forty-eight undergraduate Psychology students (8 male, 40 female, mean age: 20.1, SD = 1.7) of the University of Groningen participated volunteered to participate in the experiment in return for course credit or a compensation of €4.

Design. The experiment used a 2 × 2 design with one between (presence vs. absence of a speeded response requirement in the unconscious thought condition) and one within-subjects factor (conscious vs. unconscious deliberation). In other words, all subjects made two choices, one following unconscious and one following conscious deliberation, with half of the subjects receiving an instruction to indicate their choice as quickly as possible after the unconscious deliberation phase. The order of the two deliberation conditions was counterbalanced between subjects.

Materials. For each of the two deliberation conditions, a different set of choice options was used. One set included the four cars used by Dijksterhuis et al. (2006) and the other comprised a set of four apartments modeled after the stimuli used by Newell et al. (2009). The order in which these different choice sets were used was crossed with the order of the two deliberation conditions, thus ensuring that each choice set was used equally often in each deliberation condition.

Each of the two choice sets comprised four options and each option had twelve attributes, which could be positive or negative. The attributes specified for the four options were the same, with the only difference being whether a feature was positive (e.g., “is inexpensive”) or not (“is expensive”). The best option had 9 positive and 3 negative attributes, whereas the worst option had 3 positive and 9 negative attributes. The other two options had 6 positive and 6 negative attributes. For each choice set, the distribution of positive and negative attributes across options was fixed after an initial random assignment of values to options. The best and worst options corresponded to the first and fourth presented options for the cars and to the third and first presented options for the apartments.

Aside from their attributes, each option was assigned a fictitious name. The names of the cars were identical to those used by Dijksterhuis et al. (2006; “Kaiwa”, “Nabusi”, “Hatsdun”, and “Dasuka”). The names of the apartments included the Dutch translations of “The Pear”, “The Tulip”, “The Apple”, and “The Oak”. For each choice option, we also selected a picture of a real car or apartment building to be shown together with the attributes of the options in the information acquisition phase. Pictures were selected and evaluated in terms of homogeneity of appearance and attractiveness by the authors of this paper and a group of 6 M.Sc. students who assisted in collecting data for the experiments. Our intention was to enhance ecological validity by creating advertisement-like displays that consisted of a glossy-appearing background on which a picture of the choice option was shown together with the name of the option and the option’s attributes.

The distracting task used in the unconscious deliberation condition involved a word search puzzle that was performed on the computer. The puzzles were generated using a free online puzzle generator and each puzzle comprised a 10 by 10 array of letters that were numbered 1–100. Target words were neutral words (e.g., names of countries or vegetables), and they were shown one at a time below the letter array. The task for the subject was to find the target word in the array and to respond by means of entering the numbers that corresponded to the first and last letter. The target words could be written in any direction along a horizontal, vertical or diagonal line. Once subjects entered their response, the next target word would appear below the array, and once all words in one array had been found, the next would be presented.

Procedure. At the start of the experiment, subjects were informed that they would be performing a series of different tasks, including a choice task, a word-search puzzle, and a test involving mathematical puzzles. Next, subjects had the opportunity to practice the word-search task for one minute. This was done to familiarize subjects with the task so that they would not require additional instructions when they were presented this task during the unconscious deliberation phase. Following this practice session, the choice task began. At this point, further instructions were provided telling subjects that they would be shown a presentation of advertisements for four choice options (cars or apartments, the type of choice option was specified in the instruction) and that they should form a thorough impression of each of these options during the presentation. Upon pressing the spacebar, the presentation began and each of the twelve attributes were shown one after the other, for each of the four options, thus yielding a presentation of 48 successive displays. Before presenting the attributes of each subsequent option, an additional display was inserted that only included the name and picture of the subsequent option. This served to facilitate the distinction between different options in the sequential presentation of attributes. Each feature comprised a sentence (e.g., “the Nabusi is expensive”) shown in an advertisement-like display that also depicted the name of the option and a picture of the option. Each feature was shown for a period of 4 seconds.

Depending on the nature of the deliberation condition, subjects received different instructions following the information acquisition phase. In the conscious deliberation condition, they were shown a display in which the names and pictures of the four options were shown along with the following instruction: “You will later be asked for your opinion about the four [cars/apartments]. We now want you to think very carefully about the four [cars/apartments] and to form an opinion about them. You have three minutes to do this.” This display included a counter that indicated the passage of time in seconds. After three minutes had passed, a new display appeared that showed the four options with the numbers 1 through 4 written below them. This display included the instruction to make a choice by pressing the key that corresponded to the preferred option.

In the unconscious thought condition, subjects received the following instruction after completion of the information acquisition phase: “This is the end of the presentation. You will later be asked for your opinion about the four [cars/apartments], but you will now first have to do another task that you already practiced earlier on: Solving word-search puzzles. This time you will have three minutes to find as many words as you can. Good luck!” Upon pressing a designated key, the word search puzzle task began. After three minutes had passed, a new display appeared telling subjects to indicate their choice. In case the subject was in the group with a speeded response requirement, this instruction was: “If you were to choose one of the options now, which one would you choose? Please indicate your choice immediately by pressing the key that corresponds to the option you prefer.” The other group was asked to respond without any indication of urgency: “If you were to choose one of the options, which one would you choose? Please indicate you choice by pressing the key that corresponds to the option you prefer.”

After subjects indicated their preferred option, they were asked to indicate how important each of the attributes was to them, using a 10-point scale. After rating the attributes, subjects were shown the pictures that had been shown together with the options, and they were asked to rate the appearance of each option on a scale from 1–10. In addition, they were asked to indicate how important the appearance of the options was for their choice. On the basis of these data, we were able to derive a subjective value for each of the options that incorporated both the subjective value of the attributes as well as the value of appearance. These subjective values could then be used to determine the accuracy of subjects’ decisions in terms of whether they managed to choose the option that had the highest subjective value.

Upon completing the rating procedure, subjects were asked how much time they had spent deliberating their choice in the conscious deliberation condition and what they had done in the remaining period of the conscious deliberation phase. After typing in their answers to these queries, the subjects received instructions for the last task, the CRT (Frederick, 2005). The three items of this test were displayed on the computer monitor and subjects received a pencil and paper for making notes, if necessary. After completing the CRT, the study was done. In total, the study took approximately 30 minutes.

Data-analysis. To compare performance in the two conditions, we used three measures of choice accuracy. The first concerns the measure that has been used in most previous studies and consists of the percentage of subjects who chose the best option, that is, the option with 9 positive attributes. The second measure was the number of positive attributes associated with the chosen option. Compared to the first measure, this measure is more informative because it also distinguishes between options with 3 or 6 positive attributes. In analyzing these performance measures, we used a Wilcoxon signed ranks test to contrast performance for the conscious and unconscious deliberation conditions. The third measure of choice accuracy consisted of the subjective value of the chosen option. This measure was based on the ratings subjects assigned to the attributes and the appearance of the options. To calculate the subjective value of each option, we followed the procedure described by Newell et al. (2009). This entailed that we assigned a value of +1 to positive attributes and a value of −1 to negative attributes. For the ratings assigned for the appearance of the options, we assigned a score of −1 to options that were considered unattractive (a score of 3 or lower on the 10-point scale), a score of 0 to options considered moderately attractive (scores of 4–6), and a score of +1 to options that were considered attractive (scores of 7 and higher). Subsequently, we multiplied these values by the ratings subjects assigned for the importance of the attributes and the appearance of the options, and we summed the scores per option to derive a composite subjective value for each option. The resulting scores were compared for the conscious and unconscious deliberation conditions using a paired-samples t-test.

2.2 Results

Table 3 presents an overview of the results of Experiment 1. The percentage of subjects choosing the best option in the conscious thought (CT) condition was 52.1% whereas it was 62.5% in the unconscious thought (UT) condition. This difference did not reach significance, Z = 1.1, p = .28. Similar results were obtained in contrasting the other performance measures for the two deliberation conditions. That is, the difference in performance between the conscious and unconscious deliberation conditions also failed to reach significance when considering the number of positive attributes per chosen option and when considering the subjective value³ of the chosen options (all p’s > .36). Likewise, we also did not find evidence for an UTA in a between-subjects analysis that compared performance on the condition done first (all p’s > .77). In fact, the difference between the conscious and unconscious thought conditions was numerically greater for the conditions done second (CT = 45.8 vs. UT = 62.5% correct) than for the conditions done first (CT = 58.3 vs. UT = 62.5% correct). For the CT condition, the performance difference seen when this condition was done first vs. second failed to reach significance, p = .39.

Table 3: Results of Experiment 1. Choice accuracy defined in terms of the percentage of subjects choosing the best option, the average number of positive attributes of the chosen option, and the subjective value of the chosen option. CT denotes conscious thought whereas UT denotes unconscious thought.

Next, we asked whether the presence vs. absence of a speeded response requirement in the unconscious thought condition made a difference. Subjects on average took 9 seconds to indicate their choice in the unconscious thought condition. The average response time was significantly shorter for subjects who received the speeded response instruction, M = 7 vs. M = 11 seconds for subjects with versus without a speeded response instruction, respectively, t(46) = 2.25, p = .029. The presence of this instruction did not influence performance in the choice task, as we found that neither group showed a significant UTA on any of the three performance measures, all p’s > .32.

The assessment of how long subjects needed to their reach a decision in the conscious deliberation condition showed that the self-reported deliberation times ranged between 0 and 120 seconds, with a mean of 39 seconds. Inspection of the answers subjects provided with regard to what they did in the remaining period of the deliberation phase shows that 32 indicated they had spent this time thinking of issues unrelated to the task. The other subjects either refrained from answering or indicated that they spent some portion of this time reconsidering their decision. Interestingly, however, there was no relationship between the time subjects needed to reach a decision and the quality of this decision, with the mean self-reported deliberation durations being 37.2 and 41.4 seconds for subjects who made an incorrect versus correct choice in the conscious thought condition, p > .6.

Lastly, we examined the relationship between performance in the conscious and unconscious thought conditions, and performance on the CRT. The results obtained with the CRT showed that subjects on average answered 1.27 items correctly. To examine the relationship between CRT and the CT and UT conditions, we computed a Spearman Rank Order correlation between the number of items answered correctly on the CRT, and the number of positive attributes associated with the options chosen in the UT and CT conditions. These analyses showed that neither the correlation between CRT and CT, nor that between CRT and UT reached significance, r _s = −.2, p = .17 vs. r _s = −.14, p = .35, respectively.

2.3 Discussion

In spite of using a task that met the conditions under which previous studies found a strong UTA (Strick et al., 2011), the results of Experiment 1 did not yield a significant UTA. In addition, the results show that the presence or absence of a speeded response requirement had no significant effect on choice accuracy in the unconscious deliberation condition. On average, subjects took only 9 seconds to indicate their choice after completing the distracting task, suggesting they did not engage in extensive conscious deliberation to make this choice. By implication, it seems that our failure to replicate the UTA did not stem from the fact that subjects engaged in extensive conscious deliberation at the time of making a choice in the unconscious deliberation condition. Furthermore, our results also suggest that this failure did not result from inadvertent unconscious deliberation in the conscious deliberation condition. While our subjects did appear to use only a fraction of the available deliberation phase to contemplate their choice, the results revealed no relationship between self-reported deliberation times and choice accuracy in the conscious deliberation condition. Accordingly, the results suggest that our failure to replicate the UTA is unlikely to be due to inadvertent conscious deliberation in the unconscious thought condition or to inadvertent unconscious deliberation in the conscious thought condition.

3 Experiment 2: A role for mental fatigue?

In comparing the results of Experiment 1 to those of previous studies that did replicate the UTA, it is clear that performance in our conscious thought condition was substantially better than performance in the conscious deliberation conditions of studies that did replicate the unconscious thought advantage. Interestingly, this difference also holds across previous studies that did or did not replicate the UTA. To wit, of the experiments listed in Table 1, 26 used choice problems of the same level of complexity as we did (i.e., involving 4 options with 12 attributes each). Of these 26 experiments, 7 yielded a significant UTA and 19 produced no significant UTA. Of the seven that yielded the UTA, six yielded chance-level performance in the conscious thought condition, and only one yielded performance that was significantly better than chance. In contrast, the 19 studies that did not replicate the UTA included only one that yielded chance-level performance in the conscious thought condition, with the rest showing above-chance performance in this condition. It thus appears as if replication of the unconscious thought advantage hinges on chance-level performance in the conscious thought condition. What is left unresolved then, however, is why studies using choice problems of similar complexity (i.e., involving the same number of options and attributes; Dijksterhuis & Nordgren, 2006) yielded such different levels of performance. In other words, how can it be that the subjects in studies such as the one by Dijksterhuis et al. (2006) performed at chance in choosing the best of four cars, while the subjects in our and other studies performed well above chance on the same decision problem?

Considering that most of the experiments at stake used similar methods and subject populations (i.e., first-year undergraduate psychology students; in our case even in the same country with the same level and type of background knowledge), we arrived at the conclusion that the performance differences in the conscious thought conditions must be driven by some aspect of the experimental setting. In considering what aspects of this setting might be of importance, the results of Experiment 1 presented an interesting suggestion in showing that subjects who first completed the unconscious thought condition performed slightly worse in the conscious deliberation condition than subjects who started with the conscious deliberation condition. This alerted us to the theoretically interesting possibility that perhaps some form of mental fatigue might play a role in the UTA.⁴ That is, perhaps unconscious thought only produces better choices than conscious thought if subjects are in a state of mental fatigue, the idea being that mental fatigue would have a stronger negative effect on performance in the concentration and attention-demanding conscious thought condition than it has on performance in the unconscious thought condition. Of particular interest in this regard is a study by Webster, Richter and Kruglanski (1995). This study found that subjects who had first completed a lengthy final examination made poorer judgments in a subsequent impression formation task than subjects who had not first completed the examination. Specifically, the subjects who had completed the examination were found to leap to conclusions in that they showed a strong primacy effect in their judgment of the case at hand. Generalizing this result to the conscious deliberation condition, one would expect that mental fatigue would cause subjects to consider fewer attributes, thus leading to shorter deliberation times and poorer choices in the conscious thought condition. As this negative effect of mental fatigue might not influence unconscious deliberation to the same extent, this might drive the observed UTA. To test this hypothesis, Experiment 2 examined how the requirement to first perform a series of unrelated tasks for 45 minutes would influence performance in the conscious and unconscious thought conditions of the paradigm used in Experiment 1. We reasoned that, if executing these tasks would induce mental fatigue, then performance in the conscious thought condition might suffer more than performance in the unconscious thought condition, thus creating fertile ground for observing a significant UTA.

3.1 Method

Subjects. A new group of 24 undergraduate students at the University of Groningen volunteered to take part in this study in return for course credit or a financial compensation of 8 euros. The subjects included 11 males and 13 females with a mean age of 20.5 years (SD = 1.84).

Design, materials, and procedure. The primary objective of Experiment 2 was to determine whether an unconscious thought advantage would be obtained when subjects performed the choice task under conditions of mental fatigue. To this end, we replicated Experiment 1 but this time, subjects first completed four other tasks prior to doing the choice tasks used in Experiment 1. Completing these first four tasks took up the first 45 minutes of the one-hour session, leaving 15 minutes to complete the two choice tasks. Due to time constraints, we had to omit the rating procedure that was used in Experiment 1 to compute the subjective values of the options.

The tasks used to induce mental fatigue included the CRT (Frederick, 2005), an affective evaluation task, a mental rotation task, and an attentional blink task. In the affective evaluation task, subjects were shown two versions of the same abstract painting. One was the original painting and the other was a rotated version of the original that could also be mirror-reversed. The paintings were shown simultaneously to the left and right of fixation and the task for the subjects was to indicate which of the two versions they found most attractive. In addition, subjects were asked to describe the reasons for their choice. This task was done 20 times without breaks, taking approximately 10 minutes. The mental rotation task involved the same types of stimuli as the affective evaluation task, but now the task for subjects was to indicate whether the two versions of the painting were mirror reversed or not. This task too was repeated 20 times, for a total duration of 5–7 minutes. In the attentional blink task, subjects were shown rapid sequences of digits that included two letters, with the task being to report the identity of the two letters at the end of each trial. This task included a total of 180 trial sequences, and took approximately 25 minutes.

3.2 Results

In a first analysis, the results of Experiment 2 were compared to those of Experiment 1 to determine whether the requirement to first perform a number of unrelated tasks indeed led to reduced choice accuracy and deliberation times in Experiment 2. The results show that there was a trend towards a reduction in choice accuracy in Experiment 2. Averaged across the two deliberation conditions, 41.7% of the subjects chose the best option in Experiment 2 while 57.3% of the subjects chose the best option in Experiment 1, Z = 1.76, p = .08. In addition to this slight drop in choice accuracy, the comparison of Experiments 1 and 2 revealed that subjects in Experiment 2 took less time than those in Experiment 1 to arrive at their decision in the conscious deliberation condition. Specifically, while subjects in Experiment 1 indicated they needed 39 seconds to reach a decision, the subjects in Experiment 2 indicated they needed only 23 seconds, t(70) = 1.94, p = .056. Taken together, these results suggest that the requirement to first perform a number of other tasks indeed produced some degree of mental fatigue, resulting in shorter deliberation times and poorer judgments. But does this state of mental fatigue also lead to an unconscious thought advantage?

A comparison of choice accuracy in the conscious and unconscious deliberation conditions of Experiment 2 showed no evidence for an unconscious thought advantage. Instead, performance in the conscious and unconscious thought conditions was numerically equivalent, such that in both conditions, 41.7% of the subjects chose the best option, Z = 0, p = 1.00. A comparison of the average number of positive attributes per chosen option showed no significant difference either, Z = .24, p = .81.

3.3 Discussion

Compared to Experiment 1, the results of Experiment 2 showed a slight reduction in choice accuracy and deliberation time, suggesting that the requirement to first perform a number of unrelated tasks was indeed effective in producing some degree of mental fatigue. Yet, this manipulation did not result in chance-level performance in the conscious thought condition, and it also did not lead to a UTA. By implication, the results of Experiment 2 leave unresolved what it takes for performance in a conscious deliberation condition to drop to chance levels.

4 Experiment 3: A role for decision fatigue?

In Experiment 3 we continued our search by examining whether a more specific type of mental fatigue—decision fatigue – may play a role in the unconscious thought advantage. As shown in previous studies, repeatedly making choices will eventually lead to “decision fatigue”, a loss of motivation and effort, which would hinder the ability to choose the best option available (e.g., Vohs et al., 2008). Given that this effect is most pronounced when subjects need to make choices, it follows that perhaps the difference between studies that did and did not replicate the UTA was in whether subjects first performed other tasks in which they needed to make choices. The main goal of Experiment 3 was to examine this possibility. To this end, the subjects in Experiment 3 made eight consecutive choices, alternating between conscious and unconscious deliberation conditions. According to the decision fatigue hypothesis, we predicted a decline in choice accuracy across the four conscious deliberation conditions, but not for the unconscious deliberation conditions, eventually resulting in an UTA.

Asides from examining whether the UTA would surface in a task in which subjects made a series of eight complex choices, instead of just one or two, Experiment 3 also addressed an auxiliary issue, namely the relationship between memory and performance in the conscious and unconscious thought conditions. Since accurate performance on the choice tasks requires that a large amount of information is encoded in memory during the information acquisition phase, it would seem as though people with better memory might perform better on the choice tasks. To address this question, we had the subjects complete a basic verbal memory test which required recall of fifteen words that were shown sequentially at a slow pace (2 seconds per word), thus matching the sequential presentation of attributes in the choice tasks. This memory test was done twice with two sets of words, and we examined the correlation between the average performance on these memory tests and the average performance for choices made in the conscious and unconscious thought conditions.

4.1 Method

Subjects. A new group of 32 undergraduate Psychology students from the University of Groningen took part in Experiment 3 in return for course credit. The subjects included 9 males and 23 females (mean age = 19.3, SD = 1.3).

Design, materials, and procedure. The design, materials, and procedures used in the choice tasks in Experiment 3 were identical to those used in the choice task in Experiment 1, with the following exceptions. To start, in Experiment 3, each subject made eight choices. Four choices were made after conscious deliberation, and the other four were made after solving word-search puzzles for a period of three minutes. An important difference with the previous experiments was the use of a self-paced deliberation phase in the conscious deliberation condition. That is, in Experiment 3, subjects could indicate their choice once they had made up their mind and at this point the programme would continue with the next task. The conscious and unconscious deliberation conditions alternated across the eight-choice sequence and half the subjects started with a conscious deliberation condition, while the other half started with an unconscious deliberation condition.

The choice sets used in Experiment 3 included the cars and apartments used in the previous experiments, and two newly constructed choice sets. The first of these comprised a set of four laptops. The laptops were specified by attributes that were selected on the basis of a survey in which a different group of students indicated which attributes they consider to be most important in a laptop. The second new choice set consisted of four roommates, again specified by twelve attributes. For the roommates, the selection of attributes was based on discussions with a group of 6 M.Sc. students who assisted in the collection of data for the current studies. As in the previous experiments, a picture depicting the option accompanied the presentation of information about the choice options. For the laptops, these pictures were selected from an online image-search. For the roommates, we used the face-averaging application available through www.faceresearch.org to create four average faces that each comprised the faces of 16 different individuals. By using average faces, we ensured that the faces were approximately equal in attractiveness (e.g., Langlois & Roggman, 1990).

For each of the four choice sets, we made two versions that differed in names assigned to the options, and the pictures used to depict the options and in terms of how the positive and negative attributes were distributed across the options. In distributing the eight choice sets across the eight conditions, we used the same order for choices 1–4 and 5–8. In other words, if a subject encountered cars, apartments, roommates, and then laptops for choices 1-4, this subject would encounter the same sequence of choice sets for choices 5–8. The order in which the four choice sets were presented was determined using a Latin square. This ensured that each choice set was used equally often for each of the eight choices. Lastly, we controlled and counterbalanced the order in which the two versions of each choice set were used. This was done to preclude any systematic effect of choice set on choice accuracy.

Asides from the choice tasks, Experiment 3 also included a verbal memory test to examine the relationship between memory and performance on the choice tasks. The memory tests involved the sequential presentation of a list of 15 common words that were unrelated to the stimuli used in the choice tasks. Each word was presented for 2 seconds, and at the end of the presentation, subjects had to report as many words as they could, by typing them in on the keyboard. This test was done twice, with two different sets of words.

The order of the different tasks was fixed. As in the previous experiments, subjects first practiced the word-search puzzle task. This was followed by a memory test, the eight choice tasks, the second memory test, and then the CRT (Frederick, 2005). The experiment took one hour to be completed.

4.2 Results

The primary question of interest was how the requirement to make a series of choices would influence performance in the conscious and unconscious deliberation conditions. To address this question, we first examined choice accuracy as a function of condition (conscious vs. unconscious) and choice serial position (1–4). The results, shown in Table 5, revealed little effect of condition and serial position. In fact, most choices were accurate, and, accordingly, the average number of positive attributes per chosen alternative was close to the maximum of 9. The apparent lack of an effect of condition and serial position was confirmed by the results of an analysis that used a generalized estimating equations procedure to examine the relationship between choice accuracy, and the within-subjects factors choice serial position (1–4), and choice condition (UT vs. CT). This analysis revealed no significant effects on choice accuracy or on the average number of positive attributes of the selected options, all p’s > .26.

Table 5: Results averaged across the four experiments. The table indicates what percentage of subjects chose the best option (% Best Option) and what percentage of subjects showed a CTA, UTA, or equal performance in the conscious (CT) and unconscious (UT) thought conditions (% Subjects). A CTA was said to be present if the option chosen in the CT condition had more positive attributes than the option chosen in the UTA condition (and vice versa for the UTA).

Another matter of interest regarded the time taken to make a choice in the conscious deliberation condition. Unlike the two previous experiments, Experiment 3 used a self-paced conscious deliberation condition, meaning that subjects could enter their choice once they had made up their mind. The average deliberation time across the four choices in the conscious deliberation condition was 21.2 seconds. An analysis examining how deliberation times varied across the four choices in this condition showed a significant decline in deliberation time across these four choices, with the mean deliberation times for choices 1 through 4 being 33.0, 22.5, 20.1, and 13.1 seconds, respectively, F(1, 31) = 39.13, p < .001. Thus, while the analysis of choice accuracy showed no differences across the series of four choices, the deliberation times did show a significant decline.

Lastly, we examined the performance on the two memory tests and their relationship with performance on the choice tasks. For the first memory test, the mean number of words recalled was 6.7 (SD = 1.0), and for the second it was 7.8 (SD = 2.2). A paired-samples t-test showed that the difference in performance on the two tests was significant, t(31) = 2.61, p = .014, possibly reflecting a training or familiarization effect as the subjects knew what to expect when they encountered this task for the second time. To investigate the relationship between performance on the memory tests and choice tasks, we computed the average score on the two memory tests and we ran a Spearman’s Rank Order correlation analysis to determine the relationship between this measure and performance in the conscious and unconscious thought conditions. The results revealed a significant positive correlation between performance on the memory tests and performance in the unconscious thought condition, with r _s = .431, p = .014 for the correlation with choice accuracy and r _s = .391, p = .027 for the correlation with the number of positive attributes of the chosen option. No such correlation was found between the memory test and performance in the conscious thought condition, both p’s > .78. Using a Steiger’s Z-test for dependent correlations (Steiger, 1980), we found that the correlations between memory performance and performance in the conscious and unconscious were significantly different, Z = 4.67, p < .01.

4.3 Discussion

The results of Experiment 3 show that, across a series of choices, the unconscious and conscious thought conditions continue to produce the same level of performance, without evidence for an UTA. Assuming that making a series of complex choices would induce some degree of decision fatigue, this result can be interpreted as evidence that this factor does not play a crucial role in the UTA. Furthermore, the fact that performance in both conditions remained stable across a series of choices is of significance because it suggests that behavior in the unconscious thought paradigm does not change with repeated testing. In this regard, the results of Experiment 3 further validate the use of a within-subjects design in research on this task, as these results make clear that performance in the conscious and unconscious thought conditions is little affected by the kind and number of choices made earlier in the experiment.

A second finding of interest regards the time subjects needed to make a choice in the conscious deliberation condition. While our previous experiments assessed “deliberation time” by means of retrospective self-report, Experiment 3 used a self-paced deliberation phase in which subjects could indicate their choice once they had made up their mind. The results matched those obtained with the self-report measure, as the deliberation time for the first choice in the conscious thought condition in Experiment 3 was 33 seconds while the average self-reported deliberation time in Experiment 1 was 39 seconds. These findings converge with the results reported by Payne and colleagues, who also used a self-paced conscious thought condition and found deliberation times of 49 and 33.5 seconds (Payne et al., 2008). Taken together, these findings show that the use of a fixed deliberation phase of three or even more minutes is unwarranted because it is unnessecarily long. Furthermore, it seems as if the use of such a long deliberation phase could have an undesirable, detrimental effect on performance, as the study by Payne and colleagues found that performance was better when the conscious thought condition was self-paced rather than fixed and of unnecessarily long duration. We found a similar effect in an exploratory analysis that compared performance in Experiment 1 with performance for the first choices made in Experiment 3 (see Table 5), including only those subjects of Experiment 1 who received the speeded response instruction in the unconscious thought condition—an instruction provided to all subjects in Experiment 3. Performance in the unconscious deliberation condition did not differ between experiments, with M = 67% and M = 72% correct for subjects in Experiments 1 and 3, respectively, Z = .42, p = .68. In contrast, performance in the conscious deliberation condition was significantly better in Experiment 3, with M = 54% vs. M = 84% correct for subjects in Experiments 1 and 3, respectively, Z = 2.45, p = .014. Taken together, these findings show that a fixed deliberation phase of three or more minutes is unnecessarily long, and they suggest that the low performance often seen in a conscious thought condition may be due to a detrimental effect of the use of a fixed and unnecessarily long deliberation phase.

A last finding worth elaboration and discussion regards the positive relationship between performance in the unconscious thought condition and performance on an unrelated memory task, a relationship not found for performance in the conscious thought condition. Taking performance in the memory tests as an index of the subjects’ ability to memorize and remember information, the lack of a relation between performance in the memory test and the conscious thought condition can be taken to suggest that people who are better at memorizing and remembering information are not necessarily better in using a large amount of earlier presented information in making a choice. On the other hand, the fact that memory performance did show a positive relation with performance in the unconscious thought condition suggests that people who are better at memorizing and remembering information are also better able to choose the best option after executing the distracting task. One explanation for this result could be that, during the execution of the distracting task, memory for the earlier presented information will suffer interference from the concurrent task, a well-known phenomenon in research on memory retention (e.g., Barouillet, Bernardin, & Camos, 2004; Borst, Taatgen & Van Rijn, 2010). Perhaps the rate at which such forgetting occurred was slower for people with high scores on the memory test, thus allowing them to make better choices than people scoring low on the memory tests. Further research will be required to address this matter.

5 Experiment 4: Effect of choice complexity

While the previous two experiments examined whether the inconsistent results obtained in the unconscious thought paradigm might stem from situational aspects such as decision and mental fatigue, Experiment 4 sought to address a possibility that stems directly from the core of the DWA hypothesis: the idea that unconscious thought will produce better choices than conscious thought only if the choice problem is sufficiently complex. As alluded to earlier, an interesting fact about the studies that failed to replicate the unconscious thought advantage is that all of them – including ours—showed relatively accurate performance in the conscious thought condition. An important implication of this fact is that these replication failures thus do not provide a conclusive argument against the DWA hypothesis. After all, this hypothesis holds that unconscious thought will lead to better judgments only in case of decision problems that are sufficiently complex to surpass the limited capacity of conscious thought (Dijksterhuis & Nordgren, 2006). Hence, if we take performance in the conscious thought condition as a valid indicator of choice complexity, it follows that the DWA hypothesis could explain these replication failures in terms of the choices not having been sufficiently complex, for whatever reason. In Experiment 4, we examined this core prediction of the DWA hypothesis by increasing the complexity of the choices we used in the earlier experiments. To this end, we increased the number of options to 5 instead of 4, and we specified 15 attributes per option, instead of 12. Furthermore, we increased the difficulty of the choices by reducing the difference in how many positive attributes each option had. Specifically, the five options had 6, 7, 8, 9, or 10 positive attributes.

5.1 Method

Subjects. The sample of subjects in Experiment 4 consisted of a group of 24 undergraduate Psychology students from the University of Groningen who had not taken part in any previous experiment contrasting conscious and unconscious deliberation. Subjects were 10 males and 14 females (mean age = 19.8, SD = 4.6) who volunteered to take part in the experiment in return for course credit or a financial compensation of €4.

Design, materials, and procedure. The design, materials, and procedures of Experiment 4 were identical to those used of Experiment 1, with the following exceptions. Firstly, we included a fifth choice option in each of the two choice sets. For the cars, this was a car named “Akira”, while for the apartments, this was an apartment names “The Beech” in Dutch. For each of these options, a new picture was selected online that matched the general appearance and attractiveness of the pictures used for the other four options. A second change to the materials used in Experiment 1 was that in Experiment 3 each option had 15 attributes. These attributes included the twelve used in Experiment 1 and three newly added attributes. For the cars, the three newly added attributes included whether or not the car had a built-in navigation system, an automatic car locking system, and air-conditioning. For the apartments, the three newly added attributes specified the number of roommates one had to share the bathroom with (many or few), whether the landlord was helpful, and whether the upstairs neighbors were noisy. The distribution of positive attributes across the choice options was such that cars 1-5 had 10, 8, 9, 6, and 7 positive attributes, respectively, while for the apartments the five options had 6, 8, 10, 9, and 7 positive attributes, respectively. In other words, for the set of cars, the first car was the best option while for the set of apartments the third option constituted the best option.

5.2 Results

Of the 24 subjects included in the experiment, 45.8% chose the best option in the conscious deliberation condition while only 20.8% chose the best option in the unconscious deliberation condition. Performance in the conscious deliberation condition was significantly better than chance, χ²(1) = 10.01, p = .002, but performance in the unconscious deliberation condition was statistically equivalent to the chance level of 20%, χ²(1) = 0.10, p = .92. The difference between choice accuracy in the conscious and unconscious deliberation conditions showed a trend towards a significant conscious thought advantage, Z = 1.7, p = .08. The comparison of the number of positive attributes associated with the options chosen reflected the same trend, Z = 1.9, p = .053.

5.3 Discussion

The results of Experiment 4 show that, when choice complexity was increased (relative to Experiment 1), performance dropped to chance level in the unconscious deliberation condition, but not in the conscious deliberation condition. Furthermore, the difference in performance between the deliberation conditions approached significance, suggesting that the increase in choice complexity led to a conscious thought advantage. Taken together with the fact that our earlier experiments using less complex decisions showed equal performance in the conscious and unconscious thought conditions, this tentative finding poses a seemingly insurmountable challenge to the DWA hypothesis. Notably, this combination of results suggests that increasing choice complexity leads to worse performance in the unconscious thought condition, with the result being a conscious thought advantage—the direct opposite of what the DWA hypothesis would predict.

6 Individual differences: An analysis across Experiment 1–4

Taken together, the results of the current experiments suggest that the occurrence of a significant UTA does not depend on the presentation format, complexity, or context in which choices need to be made. Accordingly, it seems as if the occurrence of a UTA does not depend on the nature of the task. In the following section, we report an analysis that asks whether the UTA might be specific to certain subjects. To this end, we ran an analysis that incorporated the data from all four experiments, and examined what percentage of subjects showed a UTA, and what characteristics set these subjects apart from those who did not show this effect. In addition, we examined if and how performance in the conscious condition is related to performance in the unconscious thought condition.

The data used for the analysis across experiments are shown in Table 4. The table lists what percentage of subjects chose the best option in the conscious and unconscious thought conditions, and what percentage of subjects showed a CTA, UTA, or equal performance in these conditions. The latter measure was based on how many positive attributes were associated with the options a subject chose in the conscious and unconscious thought conditions. If the option chosen in the conscious thought condition had more positive attributes than the one chosen in the unconscious thought condition, a CTA was said to be present, and vice versa for the UTA. For Experiment 3, where subjects made four choices in each condition, we used the average choice accuracy and the average number of positive attributes.

Table 4: Results of Experiment 3. Choice accuracy defined in terms of the percentage of subjects choosing the best option (“% Best Option”), or the average number of positive attributes of the chosen option (“# Positive Attributes”), for the four choices made in the conscious (CT) and unconscious thought (UT) conditions of Experiment 3.

Next, we asked whether any of the factors we assessed in each these experiments could predict who showed an unconscious thought advantage, and who did not. To this end, we submitted the data from all four experiments to a multinomial logistic regression analysis that incorporated Experiment (1–4), CRT performance, the time needed to reach a decision in the conscious thought condition, and gender (male vs. female) as possible predictors. Performance on the CRT and Z-scores for deliberation times were included as covariates. We only examined the main effects of these factors.

The model was found to fit the data better than an intercept-only model, χ²(6) = 23.45, p = .001. Of the factors included in the model, gender was found to be a significant predictor of who showed equal performance, or a CTA or UTA, χ²(1) = 12.0, , p = .001 (β = 1.47 with SE = .42). As can be seen in Table 5, male subjects were more likely to show a CTA while female subjects were more likely to show a UTA. Indeed, follow-up tests showed that male subjects performed significantly better in the conscious than in the unconscious thought condition, with Z = 2.3, p = .021 for the comparison of choice accuracy and Z = 2.5, p = .012 for the comparison of how many positive attributes the chosen options had. In contrast, female subjects showed a trend in the opposite direction, with Z = 1.3, p = .16 for the comparison of choice accuracy and Z = 1.7, p = .09 for the comparison of how many positive attributes the chosen options had. Except for the gender effect, none of the other effects included in the regression model reached significance, all p’s > .19. In other words, performance on the occurrence of a UTA or CTA was independent of performance on the CRT and the time needed to reach a decision in the conscious thought condition.

In a further exploration of the gender effect, we compared the other available measures for male and female subjects. The results for the comparison of performance on the cognitive reflection test showed that male subjects performed significantly better on the cognitive reflection test, with M = 1.65 vs. M = 1.14 items correct for male and female subjects, respectively, Z = 2.60, p = .009. This finding replicates the results reported by Frederick (2005) and suggests that male subjects were more likely than female subjects to move beyond intuition and to apply analytical thinking to answer the items on this test. When taken at face value, this finding could be taken to suggest that the relationship between gender and conscious thought performance was driven by a difference in analytical thinking. Importantly, however, a further analysis showed this was not the case as there was no relationhip between performance on the CRT and performance in the conscious thought condition⁵, r _s = .041, p = .649. The second measure compared between male and female subjects was the time taken to think about the choice options in the conscious deliberation condition, a measure that was based on the time taken to indicate a choice in Experiment 3, and on self-report in the other experiments. The comparison showed a trend for males to take longer than females to think through their choice in the conscious thought condition, M = 39 vs. M = 26 seconds, respectively, F(1, 125) = 2.51, p = .12.

In a final analysis, we used the data across the four experiments to examine the relationship between performance in the conscious and unconscious thought conditions. To this end, we ran a Spearman Rank Correlation test on the number of positive attributes associated with the options chosen in the two conditions. For Experiment 3, we included only the results for the first conscious and unconscious thought choice. The results of the analysis revealed a weak but significant positive correlation between performance in the conscious and unconscious thought conditions, r _s = .183, p = .038. In other words, subjects who performed well in the conscious thought condition also performed well in the unconscious thought condition.

6.1 Discussion

The results of the analysis across experiments show that, while most of our subjects performed equally well in the conscious and unconscious thought conditions, the percentage of subjects who showed a UTA was slightly lower than the percentage of subjects who showed a CTA. Interestingly, our exploratory analysis signaled a significant contribution of gender in predicting whether subjects showed equal performance, a CTA or an UTA. Of our male subjects, 55% showed a CTA while only 17.5% showed a UTA. In contrast, only 21.6% of our female subjects showed a CTA while 31.8% of them showed a UTA. Whether or not a subject showed a CTA or UTA was unrelated to performance on the CRT, a test of intuitive and reflective thinking, and it was also unrelated to the time taken to make a choice in the conscious deliberation condition. Lastly, a correlation analysis revealed a significant positive relationship between performance on the conscious and unconscious thought conditions, indicating that subjects who performed well in the conscious thought condition were also likely to perform well in the unconscious thought condition.

7 General discussion

In the current study, we sought to identify the conditions that are required for replication of the UTA. In our first experiment, we used a task that met the conditions under which the unconscious thought advantage was strongest in previous studies, according to a recent meta-analysis by Strick et al. (2011). The results did not show a significant unconscious thought benefit, thus leading us to examine a number of additional factors that could be of importance for replicating the UTA. Experiments 2 and 3 examined performance under conditions intended to induce mental and/or decision fatigue. The results of these experiments again failed to replicate the UTA, suggesting that fatigue does not play a crucial role in obtaining the UTA. In our fourth and last experiment, we turned to a core prediction of the DWA hypothesis, which is that the UTA will only be obtained in case of a sufficiently complex decision problem. To test this prediction, we increased choice complexity by including more options and attributes. In contrast to DWA’s prediction, the results showed a conscious thought advantage.

In accounting for the fact that each of our experiments failed to replicate the UTA, a first question that may be asked regards whether our failure to replicate the UTA might have derived from the use of a within-subjects design. After all, previous studies that did find a significant UTA all used a between-subjects design and a within-subjects design could lead to different results because in this design, each subject makes two choices, and the experience of the first choice could influence the way in which the second is dealt with. Our data provide two arguments against this possibility. To start, performance and the outcomes of statistical tests were comparable across our between and within subject analyses, suggesting that choice of design has little effect on task performance. Secondly, the results of Experiment 3 showed that performance remained stable across a series of eight choices that alternated between conscious and unconscious thinking conditions. This suggests that repeated exposure to the task does not influence performance, and thus further validates the use of a within-subjects design. Taken together, these findings indicate that it is unlikely that our failure to replicate the UTA was due to the fact that our within-subjects design led subjects to execute the task differently than would be the case in a between-subjects design.

Indeed, if anything, our results make clear that a within-subjects design should be preferred over a between-subjects design. Our results show that there are individual differences in performance in the unconscious and conscious thought conditions. Specifically, our analyses show that, while 41% of our subjects made choices of equivalent quality in the conscious and unconscious thought conditions, 32% showed a CTA and 27% showed a UTA. By implication, it seems as though some people make better choices based on conscious thought whereas others make better choices after first executing a distracting task. Interestingly, we found that gender was a predictor of who showed a CTA or UTA, as male subjects were more likely to show a CTA and female subjects were more likely to show a UTA. Taken at face value, this result might be taken to suggest that the UTA could be specific to female subjects. Importantly, however, the performance difference seen for female subjects was only slight, and non-significant, suggesting that, even if our study had included only female subjects, the effect would still not be as large as that observed in the previous studies that did replicate the UTA. Furthermore, the gender effect needs to be interpreted with caution for two other reasons. Firstly, although it was highly significant in one analysis, it was obtained in a post-hoc, exploratory analysis and therefore begs for further empirical corroboration. Secondly, the direction of the gender effect conflicts with the results of the sole previous study that incorporated gender as a factor in the comparison of conscious and unconscious thought performance. Notably, Dijksterhuis (Experiment 3; 2004), found that male subjects showed a significant UTA while female subjects showed no such effect. The reliability of this finding is questionable, however, as it was seen in only one of the three experiments reported by Dijksterhuis (2004), and because it was based on a comparison of two groups of males of unknown, but seemingly small size.⁶ Nevertheless, the current findings and those reported by Dijksterhuis (2004) make clear that future studies would benefit from controlling for gender in comparing performance between a conscious and unconscious deliberation condition. Since there may be many more sources of individual differences in performance in the conscious and unconscious thought conditions, including a possible role for memory in unconscious thought (as suggested by Experiment 3), it seems that the best approach for controlling for these differences would be to use a within-subjects design.

7.1 The role of a fixed deliberation time

In the course of our attempts at replicating the UTA, we assessed and manipulated a number of factors potentially of interest to the interpretation of choice accuracy in the conscious and unconscious deliberation conditions. A central question of interest in interpreting the results of these conditions regards the time subjects take to think through their choice, both in the conscious and unconscious thought conditions. In the unconscious thought condition, the requirement to execute the distracting task is assumed to prevent conscious deliberation, but this does not preclude the possibility that subjects will still engage in conscious thought at the time of making their choice, after the distracting task has been completed. To address this possibility, we assessed response times in the unconscious thought condition, and we asked whether choice accuracy depends on the presence of a speeded response requirement. The results showed that subjects took only 9 seconds to indicate their choice, and they showed that the presence or absence of the speeded response instruction did have an effect on response times, but not on choice accuracy. These results make clear that subjects do not engage in extensive, inadvertent conscious deliberation in the unconscious thought condition. Instead, subjects seem to be able to readily indicate their choice after completing the distracting task.

For the conscious deliberation condition, the question of how much time subjects spend deliberating their choice pertains to the deliberation phase in the task. Typically, this period is fixed and of 3 to 8 minutes duration, the assumption being that subjects spend this entire period—or at least most of it—thinking about the options at hand. Strikingly, however, the use of such a long deliberation phase appears to be a convention that lacks a proper empirical basis. Notably, thus far only one study has examined how much time subjects might need to think through their choice, and the results of this study suggested that 33–49 seconds would be enough (Payne et al., 2008). In the current study, we addressed this matter by asking subjects, at the end of the experiment, how long they thought they had needed to make a choice, or by recording response times in a self-paced conscious thought condition. Consistent with the results reported by Payne et al., our results showed that subjects needed only about half a minute to judge the options at hand, thus indicating that the use of a minutes-long deliberation phase is unwarranted. More importantly, our data also showed that the use of such a fixed and unnecessarily long deliberation phase is detrimental to choice accuracy, as performance was significantly better in a self-paced conscious thought condition, an effect also observed by Payne and colleagues. A possible explanation of this effect is that the use of a fixed and unnecessarily long deliberation phase might lead some subjects to doubt their initial judgment.⁷ Specifically, the instruction to think carefully about the options for three minutes could reduce people’s confidence in an already established intuitive preference, and this could lead them to choose a different, inferior, option than the one they had in mind originally (as suggested by Simmons & Lerner, 2006). While testing this hypothesis will require further research, it is clear for now that the available data show there is no unconscious thought benefit when the deliberation phase is self-paced, as opposed to fixed, and unnecessarily long.

7.2 Implications for the DWA hypothesis

Taken together, the current findings add to growing concern about the validity of the DWA hypothesis and they shed doubt on the internal validity of the claim that performance differences in the conscious and unconscious thought conditions reflect the outcomes of a conscious and unconscious thinking process. The validity of the DWA hypothesis can be questioned on two grounds. To start, the current experiments add to a growing number of reports that each failed to replicate the UTA in spite of using conditions under which this effect would be expected to occur. This entails that the DWA hypothesis is underconstrained as it fails to capture the essence of the conditions under which an unconscious thought advantage occurs. A second argument against the DWA hypothesis regards its core assumption that unconscious thought will produce better choices than conscious thought only if the choice problem is sufficiently complex to surpass the limited capacity of conscious thought. According to this view, the probability of observing the UTA should increase as choice complexity is increased because performance in a conscious thought condition would suffer while performance in an unconscious thought condition would not. In contrast to this prediction, our results showed that increasing choice complexity had a negative effect on performance in an unconscious thought condition, but not in a conscious thought condition. The upshot of these effects was that our results revealed a conscious thought advantage for choices of relatively high complexity, a finding opposite to DWA’s core prediction.

Asides from questioning the core assumption of the DWA hypothesis, the current findings also raise serious concerns about the interpretations commonly ascribed to performance in the conscious and unconscious thought conditions. Notably, according to the DWA hypothesis, the two conditions reflect the outcome of two different thought processes, one conscious, the other unconscious. In contrast to this view, we found that performance in the conscious and unconscious deliberation conditions showed a significant positive correlation, suggesting that a significant part of the variance in the two conditions reflects a common underlying process or ability. In addition, the current findings also question the extent to which choice accuracy in the conscious thought conditions used in previous studies indeed reflected the ability to select the best option on the basis of conscious thought. Specifically, our results, together with those reported by Payne and colleagues (2008) suggest that the use of a fixed deliberation phase of several minutes is unnecessary and detrimental to performance in the conscious thought condition. Thus, performance in a typical fixed-length conscious thought condition need not reflect the best a subject could do based on conscious thought.

8 Conclusions

In conclusion, although the current study was unsuccessful in identifying the conditions under which the UTA can be reliably obtained, it does raise some important considerations for future studies addressing the DWA hypothesis. Specifically, the current study suggests that such future studies would benefit from using a within-subjects design as there are various differences between subjects that could confound the comparison of conscious and unconscious thought in a between-subjects design. Additional recommendations include using a self-paced deliberation phase instead of a fixed and unnecessarily long deliberation phase, and an instruction that prevents subjects from engaging in extensive deliberation in the unconscious thought condition. Through incorporating these conditions, undesirable sources of variance may be constrained, and this will enhance the internal validity of conclusions derived from this paradigm. The prediction that can be drawn from the current study is that, when these sources of variance are constrained, the UTA will not be found.