Cognitive miserliness in argument literacy? Effects of intuitive andanalytic thinking on recognizing fallacies

Abstract

Fallacies are a particular type of informal argument that are psychologicallycompelling and often used for rhetorical purposes. Fallacies are unreasonablebecause the reasons they provide for their claims are irrelevant orinsufficient. Ability to recognize the weakness of fallacies is part of what wecall argument literacy and imporatant in rational thinking. Here we examineclassic fallacies of types found in textbooks. In an experiment, participantsevaluated the quality of fallacies and reasonable arguments. We instructedparticipants to think either intuitively, using their first impressions, oranalytically, using rational deliberation. We analyzed responses, responsetimes, and cursor trajectories (captured using mouse tracking). The resultsindicate that instructions to think analytically made people spend more time onthe task but did not make them change their minds more often. When participantsmade errors, they were drawn towards the correct response, while respondingcorrectly was more straightforward. The results are compatible with“smart intuition” accounts of dual-process theories of reasoning,rather than with corrective default-interventionist accounts. The findings arediscussed in relation to whether theories developed to account for formalreasoning can help to explain the processing of everyday arguments.

1 Introduction

It can be argued that good argumentation and critical thinking are central aspects of our everyday reasoning, and crucial abilities for democracy and society. Yet, the ability to distinguish well-supported informal arguments from poor arguments, i.e., argument literacy, and the cognitive underpinnings of this ability, have not received recognition as central topics for mainstream cognitive psychology. Calls to bring the attention of psychologists to informal everyday argumentation have been made for years (e.g., Hornikx & Hahn, 2012; Kuhn, 1991; Shaw, 1996; Voss & Means, 1991; Voss et al., 1986), but so far, the research has tended to focus on precisely defined topics and particular types of arguments, rather than more broadly on argumentation as it appears in everyday discourse (Bonnefon, 2012). Thus, the cognitive processes that explain why people often trust poorly justified arguments are still largely unknown.

In everyday discourse, arguments can take many shapes, but any set of statements in which sets of reasons are given in support of a claim is an argument. Argument are strong when the reasons given to support a claim are relevant and sufficient. In weak arguments, the reasons given may be irrelevant, insufficient, or misleading.

The classic fallacies make up a particularly salient type of weak argument. These are arguments that turn out to be weak if inspected closely, but that are psychologically compelling on the surface. Thus, they are often used strategically and rhetorically. Textbooks on argumentation and critical thinking list many of these typical errors of argumentation that even educated and intelligent people typically make and fall prey to (Hamblin, 1970; Tindale, 2007). For example, the fallacy of a false dilemma is common in political debates, which present topics as if only two options were possible: immigrants are presented either as innocent victims or as opportunists taking advantage of their hosts. Another is the slippery slope fallacy, which argues against actions by claiming that they would lead to a chain of events culminating in terrible outcomes: “You should not let the authorities impose lockdowns even during a pandemic, because doing so will lead to fascism.”

There is largely consensus on what is erroneous in each famous type of fallacy (e.g., Hamblin, 1970; Tindale, 2007), although debates continue (e.g., Fumagalli, 2020). More detailed analyses of particular types of fallacies have been presented at least in a Bayesian framework (Hahn, 2020; Hahn & Oaksford, 2007) and in the framework of the pragma-dialectical rules of fair discussions (van Eemeren et al., 2009). These discussions are, however, outside the scope of the present study, as our focus is on the cognitive underpinnings of the ability to distinguish the weakness of fallacies in general. Therefore, for the present purposes, we use classic textbook examples of fallacies, whose weakness we will for now take as given.

1.1 Studying informal arguments as a form of reasoning

Currently, there is little knowledge of the thinking processes that explain the appeal of fallacies. Previous psychological research on argumentation has examined, for example, how contextual factors affect people’s ability to identify fallacies. This research shows that people are better at recognizing that fallacies are problematic when they are instructed to consider the opposite perspective or engaging in dialogue, and that familiarity with fallacies helps recognize them in real examples (Mercier, 2016; Neuman et al., 2006; Weinstock et al., 2004). People also show an awareness of context: they consider fallacies to be an acceptable tactic if one is quarreling, but not if the aim of a discussion is to convince the other part (Neuman et al., 2006).

Other research examined the processing steps involved in thinking about arguments. Philosophers argue that to evaluate the soundness of an argument, one should consider first, whether the reasons given are acceptable, true and relevant in themselves. Next, one should evaluate the relevance of the reasons for the claim (Angell, 1964). Existing research indicates that people spontaneously engage in the first step, in particular evaluating the believability of the premises (Thompson & Evans, 2012; Voss & Means, 1991). However, evaluating the relationship between claims and reasons seems to be more cognitively demanding and people often avoid doing it. For example, when asked to formulate counterarguments, many people elaborate on their own preferred explanations rather than providing evidence against the original argument (Kuhn & Modrek, 2017). In another study, even when explicitly instructed to evaluate the relationship between claims and reasons, people evaluated plausibility instead (Neuman et al., 2004). Similarly in a third study, the counterarguments that people produced tended to question the truth of the premises, rather than to question how well the premises support the claims (Shaw, 1996). Performance improved when participants were asked to rate the believability of the premises and the strength of the arguments separately, indicating that people can evaluate argument strength better than they do spontaneously, if they put their mind to it.

1.2 Argumentation and dual-process theories of reasoning

In the present study we are interested in relating argument evaluation to theories of how reasoning unfolds. According to popular dual-process theories of reasoning (DPTs), decisions arise from two main types of processing: fast and autonomous intuitive thinking, and slow and deliberate reasoning (Evans & Stanovich, 2013; Kahneman, 2011; Morewedge & Kahneman, 2010). In this framework, humans are depicted as cognitive misers, who use analytic processing sparingly, “operating most often under a least-effort principle, with intellectual values too weak to support the effort that thinking deeply requires” (Kuhn & Modrek, 2017, p. 97). Many findings in research on formal reasoning have been explained as instances of cognitive miserliness. For example, on base-rate tasks, people tend to respond in line with stereotypes rather than statistical frequency information (De Neys & Franssens, 2009; Tversky & Kahneman, 1974). Similarly, people treat ratios expressed using larger numbers, such as 18/100, as if they were higher than ratios expressed using small numbers, such as 2/10, because of the intuitive salience of the numerator (Bonner & Newell, 2010). On the popular Cognitive Reflection Test, which consists of items such as “A bat and a ball cost $1.10. The bat costs $1 more than the ball. How much does the ball cost?” the initial response for many people is 10 cents, rather than 5 cents, which requires some calculation (Frederick, 2005).

In the DPT framework, falling prey to fallacies could be explained as an instance of cognitive miserliness: it might be that fallacies appeal to intuitive thinking, in which people evaluate arguments quickly and without putting much effort into evaluation. Similar sentiments have been expressed by Schellens et al. (2017), whose discussion connects argument evaluation to Petty and Cacioppo’s (1986) Elaboration Likelihood Model of persuasion, which is a form of DPT. These authors suggested that argument quality may be processed either superficially or in more detail, but they did not explicitly test predictions derived from the theory. Similarly, Thompson and Evans (2012) suggested that argument evaluation is influenced by intuition, because people’s response choices in their study were different from their verbal explanations for these same choices. However, no studies to date have directly tested whether our tendency to fall for fallacies can be explained in terms of dual-process theories.

If fallacy acceptance can be explained as an instance of cognitive miserliness, the relationship might nevertheless not be straightforward. Dual-process theories of reasoning have largely been developed based on findings that concern people’s understanding of formal logic and statistical reasoning (Kahneman, 2003; Stanovich, 2011). Meanwhile, evaluating informal arguments differs from formal reasoning in several ways. Everyday reasoning involves dealing with problems that are ill-structured and debatable, and that may have no definitive solutions (Galotti, 1989; Kuhn, 1991). By definition, informal arguments are not categorically valid or invalid, the way that formal arguments are. Rather, they may be relatively sound or unsound (Angell, 1964). Based on the ways in which everyday argumentation differs from formal reasoning, it is not clear whether the same cognitive processes that are at play in formal reasoning apply in the same ways to everyday argumentation.

An important topic of debate in DPT concerns whether correct responses require deliberative analytic processing or whether they can arise intuitively. According to long-standing default-interventionist, or corrective, accounts of DPT, initial notions that are based on heuristics, and which are therefore often wrong, are corrected by later analytic processing when necessary (Evans, 2008). That is, correct responses tend to require analytic intervention, such as using mental arithmetic on the CRT (Cognitive Reflection Test) tasks. As regards fallacies, for people to correctly recognize fallacies as being weak, their initial attraction towards these alluring arguments would typically have to be corrected through careful deliberation.

In contrast, more recent “logical intuition” models (“DPT 2.0”; De Neys, 2018) have been formulated to accommodate evidence showing that even normatively correct responses on formal reasoning tasks may in fact arise immediately and effortlessly (e.g., De Neys, 2012; Newman et al., 2017). These accounts suggest that decision-making may not always require deliberation, but may instead involve a choice between competing processes that all begin immediately. Some of these intuitions may be logical if the individual has internalized basic principles of logic to the point where these can be activated and applied effortlessly. For example, on base-rate tasks people may, through practice, have learned to immediately pay attention to the base-rates rather than to stereotypes (Bago & De Neys, 2017; De Neys & Pennycook, 2019; Newman et al., 2017).

To date, logical intuitions have been discussed only in the context of formal reasoning. For informal arguments, we can imagine that thinking about them could involve comparable well-justified intuitions. For fallacies, such “smart intuitions” would mean that people would have learned, to the point of automaticity, to recognize fallacies as being weak. For example, on arguments that appeal to authority, people would intuitively know that they should pay attention to whether the authority is relevant, because that is the relevant question to consider when evaluating an authority argument (van Eemeren et al., 2009). On arguments that appeal to consequences, people would intuitively pay attention to the plausibility of the proffered consequences, thereby intuitively recognizing most slippery slope arguments as being exaggerated, implausible and weak.

DPTs lead to several predictions regarding how people evaluate fallacies. First, a corrective account predicts that rapid intuitions first lead thinkers to accept fallacies, and slower deliberation is needed to reject fallacies. By this account, we could expect that if people are encouraged to rely on intuitive thinking, they would accept an increased number of fallacies, and to do it faster, as they would straightforwardly allow themselves to accept their first intuitions. Conversely, being encouraged to think more analytically or deliberately could be expected to have the reverse effects, as the decision would now involve an enhanced conflict between the first impression and careful, deliberative scrutiny of arguments that often results in the reasoner detecting that fallacies are weak. Thus, we could expect responses to become slower and to involve more decisional conflict and weighings between options.

If, on the other hand, as logical intuitions models would suggest, judgments about the strength of arguments are based on a competition between different types of processes that all begin immediately, we could expect that manipulations to think in either way should have less effect on how people respond, as their responses would to a larger extent have been conflicted from the outset.

To gain insight into the types of processing underlying responses, one method is the analysis of response times (RT). This approach assumes that longer latencies indicate the activation of more competing elements. Previously, Voss and colleagues have used RT in studies of argumentation, finding for example that evaluating arguments that one disagrees with is slower than evaluating arguments that are in line with one’s own opinions (e.g., Voss et al., 1993). Response times are nevertheless a “black-box” method, as they do not reveal how the decision-making process unfolds (Schneider et al., 2015). Moreover, response times may be affected by other factors besides decisional conflict, such as other sources of task difficulty. Finally, the reliability of RT may suffer as a result of speed-accuracy tradeoffs and confounding variables such as differences in how certain people want to be before they give their answers. Because RT does not tell us what it is that slows down responses, using RTs as the sole indicator of the processes involved in decision making is insufficient.

1.3 Mouse tracking and changes of mind

Another method that offers to shed insight on how decisions unfold over time is mouse tracking. Mouse tracking involves following the trajectory of the computer cursor as people make decisions. This method rests on the finding that cognitive processes such as decisional conflict and hesitation between response options continually influence motoric activity, which can be seen as deviations in the path on which people move the mouse (Freeman & Ambady, 2010; Stillman et al., 2018). Indices of decisional conflict derived from mouse tracking tend to be partly dissociable from RT, typically showing correlations around r = .40 (Stillman et al., 2018). Measures of mouse trajectories have previously been used to examine the competing influences on decisions on topics ranging from self-control and food preferences to racial attitudes (Gill & Ungson, 2018; Stillman & Ferguson, 2018).

Mouse tracking methodology has also been applied to study formal reasoning. A notable finding from these studies is that the types of results that inspired the logical intuition models have not been found in mouse tracking studies. Rather, the results from mouse tracking have conformed to corrective DPTs. In one study, Szaszi et al. (2018) studied the denominator neglect task. There were no indications that correct responses could arise intuitively. Instead, better reasoners exhibited trajectories that indicated changes of mind towards the correct response. However, the denominator neglect task is very fast-paced and may thus be a poor point of comparison for the present argument evaluation task.

For making predictions about how argument evaluation will unfold in a mouse tracking experiment, it may be more relevant to turn to findings obtained using more verbose tasks that involve more elaborate reasoning. One such study is by Travers et al. (2016), who used mouse tracking to operationalize decisional conflicts on the CRT. Even these results were most compatible with corrective DPTs, because corrective mouse movements were mostly found on trials ending in correct responses, while heuristic responses did not show attraction towards the correct option.

Another task type that also involves more verbose material and more elaborate reasoning are sacrificial moral dilemmas, which ask people whether it is acceptable to kill one to save the lives of many (utilitarian thinking) or not (deontological thinking). These tasks have been studied using mouse tracking by both Gürüay and Baron (2017) and Koop (2013). These studies also looked for mind changes of the sort that corrective DPT would predict. The overall rate of switches from one side of the screen to the other was low (around 20 % of trials), and switches were not more common on trials ending in either type of response (deontological or utilitarian). Thus, in these types of tasks it seems that people usually made their choices early, with little interference on mouse trajectories from the competing response option. These findings illustrate that DPTs may not be directly applicable to reasoning on tasks outside the formal reasoning domain.

1.4 The present study

The present study tests the applicability of DPTs to everyday argumentation and asks whether humans are cognitive misers even when evaluating informal arguments. In line with this assumption and with DPTs, we predict that prompting participants to respond in line with their first impressions will make responses quicker, while prompting people to think analytically and carefully will make responses slower. In terms of performance and mouse trajectories, different DPTs lead to different predictions.

By the corrective account, correct responses typically require that analytic processing intervenes with the outputs of intuitive processes. Thus, by this account we should expect trials that end in correct responses (e.g., rating a fallacy as being weak) to exhibit attraction towards the incorrect response (e.g., rating a fallacy as being strong)¹, followed by corrective mouse movements. Incorrect responses, in turn, should be more straightforward as they would involve no analytic intervention. Encouraging people to think intuitively should decrease corrective movements and decrease performance, while encouraging analytic thinking should increase corrective movements and increase performance.

In contrast, by an account that postulates logical (or “smart” or “sound”) intuitions, no corrective movements are necessary to respond correctly, as correct responses stem from correct intuitions. An integral part of the logical intuition account is the suggestion that people are intuitively sensitive to norms of thinking even if they end up giving the incorrect response. For example, skin conductance increases when people give incorrect responses to categorial syllogisms (De Neys et al., 2010). So far, evidence for this view has been found using multiple behavioral and physiological methods, but only on formal reasoning tasks (reviewed in De Neys, 2012). So far, no studies using mouse tracking have captured evidence for this phenomenon. If it did occur, we could expect it to mean that it is the trials that end in errors that would exhibit most attraction towards the opposite response option. Instructions to think in either way (intuitively or analytically) should have little effect on performance or on mouse trajectories.

To sum up, how people evaluate and react to fallacies and other informal everyday arguments has not previously been examined in the context of DPT or from the perspective of decisional conflict, leaving the cognitive processes involved largely unknown. The present study addresses the underlying processes by experimentally manipulating thinking to be either intuitive or analytic, all the while recording our participants’ response times and mouse movements to reveal how their decisions about arguments’ quality unfold in real time.

2 Method

2.1 Participants

Fifty-eight Finnish volunteers participated in the experiment (35 female, 21 male, 1 other, 1 NA). The mean age was 30.4 years, range 17–62, SD = 13.5 years, median = 24.0 years. Thirty-six were students, 21 working, and one’s occupation was unknown. Fifteen had completed a tertiary degree and two a doctoral degree. All were native speakers of Finnish and had normal vision. The sample was a convenience sample recruited from the wider community. To control for the possible effects of familiarity with analyzing arguments, we asked whether the participants were familiar with the topic to the extent that they for example knew what is meant by “straw-man” and “slippery slope” arguments. The majority (65 %) reported no familiarity with these concepts, while 28 % reported having ‘some’ and 7 % reported ‘much’ knowledge about argumentation analysis.

2.2 Materials

2.2.1 Arguments

The stimuli originally consisted of 63 fallacious arguments and 57 reasonable arguments. The stimuli were formulated to sound like natural conversation in terms of both content and form. The fallacies were formulated with the help of textbooks and online resources (Downes, 1995; Sagan, 1997; Tenhunen, 1998) and included the following types: ad hominem, straw man, reference to irrelevant authorities, ad ignorantiam, ad baculum, ad consequentiam, false dilemma, circular argument, slippery slope, non sequitur, statistical fallacies, mixing correlation with causation, post hoc ergo propter hoc, and unfalsifiability. The reasonable arguments were formulated to be similar to the fallacies in terms of length and grammatical form. An example of a fallacy is “Petting cats relieves stress. Hence, cat owners are more easygoing than other people”. An example of a reasonable argument is “You should not drive while intoxicated because alcohol lowers reaction speed, and driving requires a good ability to react quickly”.

To make the comparison between fallacies and reasonable arguments as clear as possible, three scholars in argumentation analysis and philosophy screened the original materials to identify ambiguous items. Independently of each other, the experts rated each argument on a five-point scale, indicating whether they found the argument to be clearly a fallacy (-2), a milder fallacy (-1), difficult to assess (0), a fairly reasonable argument (+1), or a strong argument (+2). Items that were deemed difficult to assess (0) by at least one expert, or on which the experts disagreed to the extent that one of them rated it on the opposite side of the scale midpoint from the others, were excluded from further analyses. The experts agreed that the 53 remaining fallacies were fallacies and that the 29 remaining reasonable arguments were reasonable. Appendix A presents all the arguments.

2.2.2 Manipulation check

Fourteen items from the Situation-Specific Thinking Style scale (SSTS; Novak & Hoffman, 2009), translated into Finnish, were used as a manipulation check. The participants were asked to indicate on a five-point scale (1 = very poor description, 5 = very much on point) to what extent the items described the manner in which the participant had responded after the experimental manipulation. Seven items described an intuitive style of thinking (e.g., “I trusted my hunches”, Cronbach’s α = .93) and seven items an analytic style (e.g., “I reasoned things out carefully”, α = .86).

2.3 Procedure

Upon arrival at the laboratory, participants gave written informed consent. On the computer screen, they received instructions to evaluate arguments by whether the justifications given in them were strong or weak. They were given examples of strong and weak arguments and two practice trials. See Appendix B for the complete instructions.

On each trial, an argument was displayed in the middle bottom of the computer screen. In the top left and right corners of the screen were buttons with the text “Strong” and “Weak”. The participants indicated their responses by moving the computer mouse to either button and clicking it. The argument was displayed until the participant responded. For half of the participants, Strong was on the left and Weak on the right, and for the other half, the order was the reverse. To display the next argument, participants had to click “Continue” in the bottom middle of the screen. The first 60 trials were presented in this baseline condition. Then, participants received one of two experimental manipulations. Thirty participants were instructed to think intuitively “based on their first impressions” on the rest of the trials, and 28 were instructed to think analytically “as reasonably and carefully as you can” on the rest of the trials. After these manipulations, the last 60 trials were presented as above, with an additional reminder to think intuitively or analytically after 20 trials. All instructions are found in Appendix B.

The order of presentation of the items was randomized across conditions individually for each participant. Thus, the number of fallacies and reasonable arguments presented to individuals in the baseline and experimental conditions varied randomly. The number of fallacies in the baseline condition ranged from 27 to 37.

There were four combinations of manipulation and layout (intuitive with “strong” on the left; intuitive with “strong” on the right; analytic with “strong” on the left; analytic with “strong” on the right). Participants were assigned to one of these in rotating order as they arrived at the laboratory. The participants were not informed before the experiment that mouse trajectories would be recorded.

At the end of the experiment, the participants filled in a questionnaire containing the SSTS manipulation check and background questions. The participants were debriefed and they received a voucher worth 5 euros to compensate for lost time.

2.4 Mouse tracking measures

The experiment was implemented in the MouseTracker software which presents the stimuli and records the x and y coordinates of the mouse on the computer screen 60–75 times per second as participants are responding (Freeman & Ambady, 2010). To ease data processing, we used data that was time-normalized by MouseTracker to 101 steps using linear interpolation. To measure changes of mind after the participant has given consideration to both options, we calculated what proportion of each trajectory was spent on the side of the screen opposite of the final response. This was calculated as follows: X_ratio = X_opposite/(X_same + X_opposite), where X_opposite is the sum of the X-coordinates of all the time steps during which the cursor was on the opposite side of the screen from the final response, and X_same is the sum of all the time steps during which the cursor was on the same side as where the final response was made. R code for calculations is provided in the Supplementary material. The resulting measure (X ratio) is sensitive to how far to the opposite side the trajectory reaches as well as to the proportion of time spent on the opposite side.

In addition, to capture smaller deviations from a straight line, we analyzed the Area Under Curve (AUC) between the realized mouse trajectory and an idealized straight line from the start button to the response button, which was computed by MouseTracker and is a standard measure in mouse tracking research (Freeman & Ambady, 2010). Unlike the X_ratio, a high AUC does not necessarily indicate that the cursor was ever on the opposite side.

2.5 Data processing

Because the RT distribution was highly skewed to the right (skewness value = 3.47), we analyzed the natural logarithm of response times. After logarithm transformation, RT was approximately normally distributed. The data were analyzed using R (R core team, 2021), and the psych (Revelle, 2021) and psycho (Makowski, 2018) packages for R.

3 Results

3.1 Manipulation check

The results obtained using the SSTS scales indicate that the manipulations worked as intended. Participants who were encouraged to think intuitively reported higher scores on the SSTS Intuition scale (M = 3.78, SD = .69) than participants who were encouraged to think analytically (M = 2.15, SD = .75, t(56) = 8.625, p < .001). Cohen’s d was 2.26, denoting a large effect. Conversely, scores on the SSTS Analytic scale were notably higher among those receiving the analytic manipulation (M = 4.11, SD = .52) than those in the intuitive group (M = 3.12, SD = .68, t(56) = 6.209, p <.001, Cohen’s d = 1.63). As noted by a reviewer, these results may partly reflect demand effects, as the SSTS used many of the same words as the manipulations (e.g., analytic, logical, careful vs. intuition, first impression, instinct).

3.2 Effects of intuitive and analytic manipulations

Figure 1 presents means of the studied measures before and after the manipulations. Table 1 presents descriptive statistics of the change in each of these variables from baseline to after the manipulation. Table 1 also presents the results of t-tests on the size of the change depending on the type of manipulation received. All dependent variables were positively associated with each other (all p’s < .001). As the X_ratio and AUC both describe the mouse trajectories, they overlapped substantially across trials (r = .48). Their associations with RT were smaller (X_ratio: r = .24, AUC: r = .17).

Figure 1: Means of the studied measures before and after the different manipulations. Results concerning (A) error rate, (B) response times, (C) X_ratio, (D) area under curve. Error bars are 1 standard error of the mean.

Table 1: Change from baseline in the studied variables by type of manipulation

3.2.1 Ability to distinguish fallacies from reasonable arguments

To describe how participants distinguished fallacious from reasonable arguments, we first used the error rates to calculate the d’ (d-prime, discriminability) and c (criterion) measures (Green & Swets, 1966). Based on signal detection theory (SDT), the discriminability measure indicates the ability to distinguish between response options, and the criterion measure indicates the participants’ bias to prefer one response option over the other. In the present study, discriminability describes participants’ individual abilities at distinguishing fallacies as being weak, and well-justified arguments as being strong. The c measure reflects the participants’ overall tendency to rate arguments as being either strong or weak, regardless of actual strength. Larger criterion values indicate a higher threshold for accepting an argument as being strong.

Examination of the SDT measures (Table 1) and the error rates on fallacies and reasonable arguments separately (Figure 1A) indicates that participants who were encouraged to think intuitively relaxed their criterion such that they started rating both fallacies and reasonable arguments as more often being strong. Those who were encouraged to think analytically did not seem to change the way they assessed arguments. However, these trends were not statistically significant. Overall, the error rates were very low throughout the experiment.

3.2.2 Response times

As predicted, the manipulations affected RT differently. As Figure 1B shows, responses became quicker after the intuitive manipulation. After the analytic manipulation, responses seemed to become slower. As noted by a reviewer, these results offer additional support to the validity of the manipulations.

3.2.3 Analyses of trajectories

Figure 1C shows the proportions of the trajectories spent on the opposite side (the X_ratio). As the figure shows, there was no significant difference between the effects of the two manipulations on this variable.

However, as Figure 1D shows, the two manipulations affected AUC differently. Among those encouraged to respond intuitively, trajectories became more curved. Figure 2 illustrates the curvature of the trajectories averaged over trials. Note that the actual trajectories were highly variable. Figure 3 shows the actual trajectories of two participants.

Figure 2: Average trajectories before and after manipulations. In the experiment, the response buttons were labeled “Strong” and “Weak”. In the figure, all trajectories have been turned so that the correct responses (classifying fallacies as weak and reasonable arguments as strong) end on the right, all incorrect responses on the left.

Figure 3: Illustration of the mouse trajectories of one participant before and after the intuitive manipulation (top row) and one participant before and after the analytic manipulation (bottom row).

3.3 Correct versus error trials

To test differing predictions derived from corrective and logical intuition accounts of DPT, we compared all dependent variables across trials with correct and erroneous responses. Table 2 shows descriptive statistics for these variables. T-tests showed that RT, the X_ratio, and AUC were larger for error trials than for correct trials. This indicates that when making errors, participants’ mouse trajectories exhibited more attraction towards the correct response and strayed longer into the opposite side, than they did towards the incorrect response when responding correctly. This is clearly visible in Figure 2, which shows that the average trajectories of error trials initially leered towards the correct response button, before moving to the incorrect one.

Table 2: Descriptive statistics for measures on trials ending in correct responses or errors

The error rates were too low to reliably investigate whether the intuitive and analytic manipulations differently affected the RT, X_ratio, or AUC of correct trials and error trials.

3.4 Final checks

To rule out that the results were affected by how familiar the participants were with argumentation analysis, we ran the analyses again, excluding those who reported having ‘much’ knowledge about argumentation analysis. We also inspected all dependent variables for systematic differences between the participants reporting different levels of knowledge. No systematic differences were found.

We also ran all analyses separately for fallacies and reasonable arguments, but as the pattern of results was highly similar and separate analyses indicated no further conclusions, we report the results for both types of arguments pooled (except the error rates for fallacies and reasonable arguments, which Figure 1A shows separately for clarity).

Finally, we ran all analyses with the ambiguous arguments included. The pattern of results and the conclusions were unchanged.

4 Discussion

How people distinguish fallacies as being weak arguments has large practical meaning for their ability to make reasoned choices, with implications for many areas of life. In the present study, argument literacy was examined in light of a prominent notion in the literature on formal reasoning, namely that people are cognitive misers. A cognitive miser saves effort by forgoing careful scrutiny of all relevant aspects of a task and settles for an easy answer (Toplak et al., 2011, 2014). The present paper asked whether humans are cognitive misers even when evaluating informal arguments. Studies indicate that evaluating the relationship between reasons and claims is the most demanding stage of argument analysis and one that is rarely engaged in spontaneously (Shaw, 1996). Thus, miserliness in evaluating arguments could manifest as reasoners avoiding evaluating whether the reasons support the claims, basing their responses instead on more superficial features or general believability of the presented information (Kuhn & Modrek, 2017; Shaw, 1996).

To test for miserliness in argument evaluation, we subjected our study participants to the simplest of manipulations: instructions to use either a more intuitive way of thinking, or a more analytic way of thinking. Under instructions to think in one of these ways, participants evaluated whether arguments were weak or strong. To track argument evaluation as it unfolded, we analyzed error rates, response times, and indices of decisional conflict as measured through mouse tracking. In addition, we compared the cursor trajectories on trials that ended in correctly rating fallacies as weak and reasonable arguments as strong, to trials that ended in incorrectly accepting fallacies or rejecting reasonable arguments. The research results were only partly in line with the idea of a cognitive miser.

First, response times changed as a function of thinking style manipulations in the way predicted by dual-process theories, decreasing in the intuitive condition and increasing after the analytic probe. Response times are often interpreted as indicative of the amount of conflict, with faster responses taken to be less conflicted (Bonner & Newell, 2010; Voss et al., 1993). The present RT results are compatible with this view. They could suggest that in the intuitive condition, our participants gave carefree intuitive responses, while in the analytic condition they experienced more conflict and used more reasoning. However, such a conclusion is not plausible because the increased time spent in the analytic condition had no bearing on either the responses that participants gave or on the cursor trajectories leading to these responses, as we will discuss next.

Examining the responses that participants gave makes it clear that correct responses did not require analytic processing. Participants correctly tended to see that fallacies are weak and reasonable arguments are strong even when they were under instructions to think intuitively and answering quickly. By the corrective account, encouraging people to think intuitively should have increased errors, and encouraging analytic thinking should have decreased errors (Evans, 2008). In contrast, by an account that postulates smart intuitions (De Neys, 2018), instructions to think in either way should have had little effect on performance. The results showed no significant difference in how the manipulations affected the error rates. However, the overall low error rate makes it difficult to interpret the results as supporting either account, as the lack of effects might be due to a ceiling effect.

Fortunately, the trajectory analyses offer clearer conclusions. We may note that the present findings are in line with previous findings documenting that mouse tracking can reveal processes that are not easily available to introspection and also, that mouse tracking indices are at least partly dissociable from RT (Stillman et al., 2018). While RTs may be influenced by multiple factors, research indicates that mouse paths more reliably reflect decisional difficulty arising from ambivalence towards the target and from decision changes (Maldonado et al., 2019; Schneider & Schwarz, 2017; Schneider et al., 2015).

The two measures describing trajectories that we used, AUC and X_ratio, responded differently to the thinking style manipulations. AUC grew larger after the intuitive instructions, and smaller after the analytic instructions. If we assume that AUC is an indication of the amount of decisional conflict that participants experience, this result is the opposite of what DPTs would predict. However, since these responses also became quicker, the most likely explanation seems to be that the increased AUC is an artefact brought on by decreased precision as people were moving the mouse quickly. It should be noted that AUC values increase even if trajectories stay on the side of the final response button throughout the trial. Thus, they may not be the ideal measure to capture the types of mind changes that DPTs are concerned with.

In contrast to AUC, the X_ratio may be a more appropriate measure for describing changes of mind, as the X_ratio describes the extent to which the cursor trajectory strays over to the side of the screen opposite from the final response. In the present experiment, the X_ratio was equally unaffected by both manipulations.

The most interesting differences were revealed by the comparisons between trials with correct responses and trials with incorrect responses. The corrective, default-interventionist account predicts that trials that end in correct responses should exhibit attraction towards the incorrect response, reflecting incorrect heuristics, followed by corrective mouse movements, reflecting the activation of analytic thinking processes (Evans, 2008). Incorrect responses, in turn, should be more straightforward as they would involve no analytic intervention. The present results did not conform to such a model.

The results revealed the opposite: corrective movements were more common on trials that ended in errors, as shown by both AUC and the X_ratio. This type of finding is predicted by an account that postulates smart intuitions (De Neys, 2018). By such an account, no corrective movements are necessary to respond correctly. When making errors, however, the existence of these smart intuitions, or some type of implicit monitoring for conflicts between heuristics and normative principles (Johnson et al., 2016), is assumed to influence the decision making process. The present results, in which the trajectories on error trials showed attraction towards the correct response option, could be predicted by this model. Thus, when responding incorrectly, participants seemed to have some implicit awareness that fallacies are weak, and that reasonable arguments are strong. Yet they gave in to temptation and responded that the fallacies were strong or that the reasonable arguments were weak.

Previous evidence for “logical intuition” accounts has stemmed from studies focused on formal reasoning tasks. These studies have presented evidence that for many people, some principles of good reasoning have been automatized to the point of having become intuitive (e.g., Brisson et al., 2014; Handley et al., 2011; Newman et al., 2017; reviewed in De Neys, 2012, 2018). The present study extends this existing research by showing that a similar effect also seems to apply to the evaluation of everyday arguments. Thus, even though there are no definitive correct or incorrect responses on informal arguments in the way that there are on formal reasoning tasks (Angell, 1964), comparable cognitive processes may be involved in how people solve both types of thinking tasks. This finding is theoretically important because until now, research investigating whether the same cognitive processes apply to both formal and informal reasoning has been scarce and the findings inconclusive (Thompson & Evans, 2012).

The present study is also the first to report support for smart intuitions using mouse tracking methodology. Previous mouse tracking studies with reasoning tasks have been compatible with corrective DPTs, as they have found corrective cursor movements to be related to correct responses and better reasoning ability (Szaszi et al., 2018; Travers et al., 2016). The difference between our results and those of Szaszi et al. and Travers et al. may reflect the type of task used. Travers et al. (2016) discuss the possibility that on the CRT, which they studied, the logical and other principles needed to correctly solve the tasks may not be easy to internalize. The logical intuition account is largely based on studies using logical syllogisms and base-rate tasks (De Neys, 2012). It may be easier to internalize the logical principles needed on those tasks because they are relatively simple. In this comparison, the processes needed to evaluate fallacies and reasonable everyday arguments thus seem to be more similar to the processes needed for logical reasoning than to the perhaps more complex types of processes that are needed on the CRT. This is somewhat surprising, given that informal arguments, by definition, concern problems that are debatable, and that may have no definitive solutions (Angell, 1964; Galotti, 1989; Kuhn, 1991). The present task format likely required less complex processing than is needed to evaluate everyday arguments in more naturalistic settings.

4.1 Caveats

A few methodological issues regarding this study are important to be discussed further. One is that the present was a high-achieving population, as indicated by the overall low error rate. To establish whether argument evaluation involves smart intuitions more generally, it is important to replicate the present analyses in populations who more often accept fallacies at baseline. It is possible that for people who are generally good at telling fallacies from reasonable arguments, allowing themselves to respond incorrectly is particularly difficult.

A related observation is that the context of participating in a reasoning study on university premises may have acted as an unintended manipulation to think more analytically from the outset, and unintended demand effects may have been at play (Phillips et al., 2016). Studies indicate that the potential to increase analytic thinking experimentally tends to be small in any case (Lawson et al., 2020), and the present context may have restricted it further. Therefore, more research is needed to find whether the present findings replicate in other settings, or whether more intuitive thinking may decrease accuracy, and more analytic thinking may increase accuracy on argument evaluation, if the baseline is less high. For example, evaluating arguments in more naturalistic settings may better reveal how people typically think about arguments.

Unfortunately, the small sample size and resulting low statistical power precluded more detailed analyses of the effects of background variables, such as previous exposure to argumentation terminology and concepts. Thus, studies with larger samples are needed to investigate possible moderators of the ability and threshold for recognizing argument quality. Argument evaluation likely interacts with individual characteristics of the reasoner, such as reasoners’ more stable preferences for intuitive or analytic thinking. Influences of reasoners’ characteristics are likely, because thinking styles tend to predict performance on most reasoning tasks and to affect thinking and behavior widely (Pennycook et al., 2015; Petty et al., 2009; Toplak et al., 2011). Particularly relevant for evaluating arguments, a recent study showed that understanding scientific argumentation is positively predicted by general cognitive abilities and epistemological sophistication (Münchov et al., 2019). More research is needed to determine whether this also applies to fallacies and to everyday informal argumentation more generally.

By the present results it is not clear whether the findings apply to all poorly justified arguments or only to fallacies. Fallacies are defined as being unreasonable but psychologically compelling (e.g., Copi & Burgess-Jackson, 1996). That is, fallacies are not just any type of weak argument but a very specific class of weak argument that are often used for rhetoric effect precisely because there is something in them that makes people accept them. In future research, an interesting question is how people evaluate other kinds of weak arguments. There can also be arguments that are weak in a mundane, more obvious way, without conforming to any of the famous formats known as fallacies. For example, an argument such as “The postal service in Finland is excellent since it is available to anyone” is not among the famous types of fallacies recognized in textbooks, but nevertheless cannot be considered particularly strong because the reasons given are not sufficient in relation to the strength of the claim being made.

Finally, bringing research on argumentation to the cognitive lab comes with some challenges. Argumentation is an inherently social activity, and integral to argumentation is that the audience or opponent brings up counterarguments. Some theorists argue that reasoning developed to enable argumentation with other people (Haidt, 2012; Mercier, 2016). This can be contrasted with DPTs, which largely construe reasoning as if the function of reasoning were to correct intuition when it is factually incorrect. Research shows consistently that the role that an individual holds in a debate – for or against a proposition – significantly affects the ability to recognize fallacies as being weak, and that reasoning performance on a variety of tasks improves if they are framed in the context of a dialogue (Mercier, 2016; Neuman et al., 2006). Thus, future studies could benefit from incorporating social factors into the experimental paradigms alongside the modes of thinking that were the focus of the present study, to build a more comprehensive picture of how and why some people fall for fallacies while others do not.

4.2 Conclusions and future directions

How people understand informal argumentation, which is ubiquitous in all communication, has not been studied extensively from a cognitive perspective. This study presented evidence supporting the notion that falsely accepting fallacies or rejecting reasonable arguments is more complicated than is correctly rejecting fallacies and acknowledging reasonable arguments as being strong. These findings point to fundamental similarities between how people understand informal logic and recent accounts of how they understand formal logic (De Neys, 2018). Importantly, the results were not compatible with corrective dual-process models. There were no indications that responding correctly would be laborious and require analytic intervention, or that boosting efforts to think analytically would make people more likely to change their mind after forming an initial hunch. Thus, the widespread trust in bad argumentation that is obvious outside the reasoning laboratory may to some extent not be due to people not using enough mental effort to evaluate the arguments they encounter, but to a process in which they are at least implicitly aware that the argumentation is flawed but for reasons that are beyond the present paper, they choose to accept the poor arguments and reject the strong ones.