Realists and Idealists in QCA

Abstract

The sole purpose of the enhanced standard analysis (ESA) is to prevent so-called untenable assumptions in Qualitative Comparative Analysis (QCA). One source of such assumptions can be statements of necessity. QCA realists, the majority of QCA researchers, have elaborated a set of criteria for meaningful claims of necessity: empirical consistency, empirical relevance, and conceptual meaningfulness. I show that once Thiem’s (2017) data mining approach to detecting supersets is constrained by adhering to those standards, no CONSOL effect of Schneider and Wagemann’s ESA exists. QCA idealists, challenging most of QCA realists’ conventions, argue that separate searches for necessary conditions are futile because the most parsimonious solution formula reveals the minimally necessary disjunction of minimally sufficient conjunctions. Engaging with this perspective, I address several unresolved empirical and theoretical issues that seem to prevent the QCA idealist position from becoming mainstream.

1 Introduction

The sole purpose the enhanced standard analysis (ESA) has been designed for is to avoid so-called untenable assumptions. One source of such assumptions can be statements of necessity. ¹ A researcher making the necessity claim $X\leftarrow Y$ cannot make the simplifying assumption $\neg X\rightarrow Y$ during the analysis of sufficiency. ESA is agnostic as to whether $X$ stands for an atomic condition, a conjunction, a disjunction, or any combination of those. The Qualitative Comparative Analysis (QCA) community has elaborated a set of criteria, though, that define what constitutes a meaningful claim of necessity. ESA, together with existing standards, is a tool for combining the analysis of necessary conditions with that of sufficient conditions under the constraint of often noisy social science data.

Thiem (2017) does not question the appropriateness of the core purpose of ESA: to avoid untenable assumptions. What he dismisses are the criteria that distinguish meaningful necessary conditions from meaningless supersets. By trivializing the disciplined search for meaningful necessary conditions into mindless data mining for supersets, Thiem (2017) finds plenty of necessary conditions everywhere. This, in turn, produces the alleged CONSOL effect of ESA: once all supersets are allowed to be seen as necessary conditions, usually all simplifying assumptions are classified as untenable by ESA and only the conservative solution can be obtained.

In Section 2 I spell out existing standards for the analysis of necessary conditions and in Section 3 I empirically demonstrate that no CONSOL effect exists once these standards are applied. In an attempt to preempt a predictable objection to my emphasis on existing conventions in the QCA community, I introduce the notions of QCA realists and idealists. QCA idealists deem separate searches for necessary conditions as misguided. Instead, researchers simply need to confine themselves to producing the most parsimonious solution from their truth table, because this will automatically identify all minimally sufficient conjunctions that together form a minimally necessary disjunction for the outcome (e.g. Baumgartner 2015). While formally correct within a regularity theory of causation framework, in Section 3 I address several unresolved issues that arise when this position is applied to social science theories and real data. Section 4 concludes with some broader reflections on the split between QCA idealists and realists.

2 Criteria for Necessary Conditions

Thiem’s (2017) crucial claim for producing the CONSOL effect is that scholars must declare as necessary any superset (be it a single set or a disjunction ² ) of the outcome if only it passes a predefined consistency threshold. Contrary to that, QCA realists have formulated several criteria that need to be applied during the search for necessary conditions: empirical consistency, empirical relevance, and conceptual meaningfulness. Thiem chooses to dismiss all but empirical consistency without providing convincing, or sometimes even any, arguments. In my view,—and notwithstanding Thiem’s (2017) claim to the contrary (fn 3)—the only coherent critique at ESA would be to take the position of a QCA idealist and argue that separate tests of necessity are meaningless. Since Thiem (2017) does not take a coherent idealist position and instead accepts the realist strategy of separate analyses for necessity and sufficiency, it follows that his argument must be evaluated by the rules and practices of QCA realists.

2.1 Empirical consistency

Thiem, against current practice, chooses 0.75 as the consistency threshold for necessary conditions, thus making the occurrence of supersets much more likely. Justifying his “harmonization” (p. 482), Thiem (2017) misleadingly refers to Ragin (2008) and Schneider and Wagemann (2012). These authors explicitly argue that the consistency threshold should be considerably higher (at least 0.9) for necessity relations. Nowhere in Thiem (2017) is it argued in which sense prevailing reasoning about choosing high consistency thresholds for necessary conditions is flawed and in need of being changed. Even from a QCA idealist position it sounds implausible to lower the consistency threshold to 0.75.

Even if one followed Thiem (2017) in arguing that the sole decision-making criterion for identifying necessary conditions consists in whether they are supersets of the outcome, QCA realists argue that one should rely on additional information in order to assess the consistency of the set relation. For instance, Haesebrouck (2015) proposes a more conservative consistency measure. Likewise, if a set relation displays too many deviant cases consistency in kind (Schneider and Rohlfing 2013), it should not be considered a necessary condition, even if it passes the consistency threshold.

In sum, QCA realists adhere to tougher consistency criteria than Thiem (2017) and therefore tend to identify fewer necessary conditions. This, in turn, makes the alleged CONSOL effect less likely to occur.

2.2 Empirical relevance

Thiem (2017) dismisses the so-called coverage, or relevance, score and only reports it (selectively) “for completeness” (p. 481). QCA realists, among them leading scholars such as Goertz (2006), Ragin (2006), or Mahoney, Kimball, and Koivu (2009) in turn, deem it an important ingredient for any set relational assessment. Empirically consistent supersets can be—and in applied QCA often are—empirically trivial when interpreted as necessary conditions. This can either be because the superset $X$ is so much bigger than the outcome set $Y$ and/or because $X$ is so big that there is hardly any instance of $\neg X$ , i.e. $X$ is close to being a constant. The former source of trivialness is taken care of by the coverage score (Ragin 2006). The latter can be assessed by the Relevance of Necessity (RoN) score (Schneider and Wagemann 2012) and is of particular importance when disjunctions are tested. They tend to be big sets in which most (and sometimes all) cases under study are members of. It, of course, amounts to a trivial claim to argue that a (near constant) $X$ is a meaningful necessary condition for any outcome $Y$ —not least because that $X$ would also qualify as a consistent enough superset of outcome  $\neg Y$ .

For illustration, consider Thiem’s (2017) Lipset example. In his Table 2, $D+\neg U$ is listed as a necessary disjunction for outcome $S$ , with consistency 0.96 and coverage 0.51. However, that very same disjunction is also a superset of outcome $\neg S$ , with consistency 0.95 and coverage 0.55. ³ In the presence of such simultaneous superset relations, Thiem’s position strikes me as untenable, according to which the same condition $X$ has to be declared necessary for outcome $Y$ and also outcome $\neg Y$ . Such a fallacy is avoided if existing QCA realist standards are adhered to that recommend taking the RoN score into account. Expression $D+\neg U$ has very low RoN scores both for $S$ and $\neg S$ and therefore should be rejected as a necessary condition for either outcome.

In sum, QCA realists pay attention to empirical trivialness and therefore impose tougher criteria for necessary conditions than Thiem (2017). This, in turn, makes the CONSOL effect less likely to occur.

2.3 Conceptual meaningfulness

Thiem (2017) does revert to an idealist position when he states that the identification of necessary conditions is a matter of Boolean algebra rather than social science reasoning: “[…] the formal definition of the Boolean operation of implication underlying every relation of necessity, $P\leftarrow Q\equiv P+\neg Q\equiv \neg [\neg PQ]$ , neither entails anything about the structure of $P$ , nor that some higher-order concept needs to be identified by the researcher that causally connects all disjuncts $p_{1},p_{2},\ldots ,p_{j}$ of $P$ to $Q$ , […]” (p. 480). QCA realists do not follow this particular notion of necessity and, instead, argue that each single disjunct is a SUIN condition, that is, “[…] a sufficient but unnecessary part of a factor that is insufficient but necessary for an outcome” (Mahoney et al. 2009, p. 126). This means that without naming what that “factor” is, no meaningful claim of necessity can be made. QCA realists, in other words, argue that the necessary condition $P$ is a higher-order concept and usually not known prior to the analysis. Rather it needs to be conceptualized by the researcher after empirically finding that disjuncts $p_{1},p_{2},\ldots ,p_{j}$ form a superset of  $Q$ . Further, each $p_{j}$ is conceptually connected to $Q$ , not causally; they are definitional attributes of  $P$ .

Thiem (2017) also dismisses the need for a theoretically meaningful higher-order concept by taking the QCA idealist position that “[…] in every QCA solution that exhibits perfect consistency and coverage, the disjunction of minimally sufficient conditions for the outcome is also minimally necessary for that outcome, without a unifying higher-order concept being required.” (p. 480) This formalistic argument is problematic when dealing with real social science theories and data. First, while some examples of compound necessary conditions exist, ⁴ I am not aware of any social science writing in which such a complex expression as a disjunction of minimally sufficient conjunctions is interpreted as a substantively meaningful necessary condition. Second, since the introduction of parameters of fit, very few applied QCA produce solution formulas that exhibit perfect consistency and coverage and applied QCA researchers therefore end up with disjunctions of sufficient conditions that do not qualify as necessary conditions (see Section 3 for empirical evidence). QCA realists have learned to live with this whereas the position of QCA idealists on this empirical fact is yet to be spelled out.

In sum, for QCA realists “[t]he idea that the researcher must ‘make sense’ of causal conditions identified as necessary […] is very important.” (Ragin 2000, p. 209). Because QCA realists only accept conceptually meaningful supersets as necessary conditions—be it single sets, disjunctions, or conjunctions—the CONSOL effect is less likely to occur.

2.4 Correctly interpreting the superSubset output

A point related to the conceptual meaningfulness of necessity claims is the way in which the output of the superSubset function is interpreted in Thiem (2017). Thiem erroneously treats the output of the superSubset function as the product of sums when, instead, it simply lists alternative supersets of the outcome. Whenever superSubset returns something like $A+B\leftarrow Y$ and also $C+D\leftarrow Y$ , Thiem automatically accepts both disjunctions as necessary, thus de facto treating it as $(A+B)\ast (C+D)\leftarrow Y$ . The negation of this expression— $(\neg A\neg B)+(\neg C\neg D)$ —is a bigger set than just negating one of the two disjunctions. This, in turn, means more remainders need to be blocked from the analysis of sufficiency, which, in turn, means that the CONSOL effect is more likely to occur. This is a crucial, but incorrect, move in order to manufacture the alleged CONSOL effect. From an idealist position, it is incorrect simply because the very idea of exclusively looking for non-minimal supersets makes no sense.

From a QCA realist position several issues arise. Apart from the lack of conceptual meaningfulness, consider that for virtually all of the examples used in Thiem (2017), it holds that the expression derived by first rewriting the expressions returned by function superSubset as a product of sums (as de facto done by Thiem) and then minimizing this to the simplest equivalent logical expression (not done by Thiem), yields an expression that most of the time is not a superset of the outcome anymore. The perhaps most glaring example comes from Thiem’s (2017) analysis of the data from Fischer (2015) with consistency set to 0.75. The expression formed by the eight disjunctions ⁵ rewritten as the product of sums and then minimized simply yields the empty set  $\emptyset$ .

In sum, interpreting the output of the superSubset function as if all disjunctions needed to be retained and de facto combined by logical AND is surely wrong from a QCA realist perspective (and futile from a QCA idealist perspective). Once only meaningful supersets are postulated as necessary conditions, the CONSOL effect is less likely to occur.

3 Idealists and Realists in Practice

Now that the QCA realists’ standards for analyzing necessary conditions are put in place, how would the analysis of the Lipset data and the other examples look like? Would the CONSOL effect occur, i.e. would the use of ESA unavoidably lead to the conservative solution as the only possible viable solution? And, do QCA idealists propose an alternative that works?

A QCA idealist would not perform separate analyses of necessity and sufficiency because the most parsimonious solution derived from the logical minimization of the truth table reveals a minimally necessary disjunction of the minimally sufficient conjunctions. Does this work with the examples used in Thiem (2017)?

Table 1 displays the most parsimonious solution for the Lipset example. Unlike Thiem (2017), I also report the parameters of fit, consistency and coverage. As can be seen, while each conjunction passes the consistency threshold of 0.8, the coverage value for the solution formula is only 0.714. Since coverage sufficiency equals consistency necessity (Ragin 2006), this means that the consistency of the solution formula as a necessary disjunction for outcome $S$ is 0.714. This is certainly too low for QCA realists—and QCA idealists should probably be even be stricter when it comes to inconsistencies. Whereas QCA realists conclude that there are sufficient, but no necessary conditions, the stance of QCA idealists vis-à-vis such common situations in applied QCA is not clear yet.

Table 1. Most parsimonious solution, outcome $S$ .

The Lipset example is not an exception. Using the same 21 data sets as in Thiem (2017), I derive the most parsimonious solution formula for the outcome and for its negation with consistency thresholds of 0.9 and 0.75, respectively. This yields 84 different analyses. In order to fulfill the idealist’s vision in which a separate analysis of necessity is superfluous, a causally interpretable solution must fulfill at least three criteria: the disjunction must pass the necessity consistency threshold; each conjunction must pass the sufficiency consistency threshold; and there cannot be model ambiguity (Baumgartner and Thiem 2015).

Out of the 84 analyses, only 17 (20%) pass all criteria. ⁶ This seems to show that, while from the perspective of a regularity theory of causation QCA is designed for finding a necessary disjunction of minimally sufficient conjunctions, in practice that is normally not achieved. Even for those instances in which all empirical hurdles are taken, the question still remains in which sense these complex necessity statements are supposed to relate to any existing social science theory on necessary conditions. Faced with this challenge, researcher can either abandon QCA or turn to a QCA realist approach with its separate analyses of necessity and sufficiency. This is what I am going to do now. ⁷

In his Table 2, Thiem (2017) reports eight different supersets of the outcome that all pass the consistency threshold of 0.9. ⁸ QCA realists would not accept all supersets as being necessary just because they pass the consistency threshold. They would screen these supersets to see which one(s) pass the empirical and theoretical hurdles for being meaningful necessary conditions. In a first step, it is checked if any of them contains deviant cases coverage in kind. In the example at hand, it turns out that for none of the supersets $X$ any case exists with $X<0.5$ and $Y>0.5$ (see replication material).

The next selection criterion consists in checking each superset’s empirical relevance as a necessary condition (RoN). As Table 2 shows, more than half of the conditions display low RoN scores ( ${<}0.5$ ). They are supersets of the outcome because they are very big sets. As a matter of fact, almost all disjunctions with low RoN exclusively display cases with fuzzy set membership higher than 0.5 (see replication material). Ignoring the size of a set (disjunction or atomic) is problematic. For illustration, consider the fact that disjunctions $D+\neg U$ and $\neg U+I$ , which Thiem (2017) proposes to accept as necessary for outcome $S$ simply because they pass his consistency threshold, also pass that consistency threshold for outcome $\neg S$ . This phenomenon of simultaneous supersets is caused by the big size of the disjunctions in question. QCA realists would therefore reject all supersets with too low RoN.

Table 2. Supersets, outcome $S$ .

Only four expressions pass the empirical relevance test: $G$ , $L$ , $LG$ and $D+U+I$ . They still need to be scrutinized for their conceptual meaningfulness. For the sake of illustration, suppose disjunction $D+U+I$ could stand for the higher-order concept of “modernized society”  $M$ . Assume the researcher makes the necessity claim $M\leftarrow S$ , where $M=D+U+I$ . In other words, higher-order concept $M$ is the necessary condition and $D$ , $U$ , and $I$ are SUIN conditions and functionally equivalent conceptual attributes of  $M$ . According to ESA, this necessity claim blocks all logical remainder rows that are subsets of $\neg (D+U+I)=\neg D\neg U\neg I$ .

Based on this necessity claim, the ESA solution formula is $D\neg I+UG\rightarrow S$ (Table 3). As can easily be seen, this formula is identical to the most parsimonious solution reported in Thiem (2017). In other words, despite a claim of necessity involving a disjunction, ESA not only does not produce the conservative solution. In this example, it produces the most parsimonious solution.

Table 3. ESA solution formulas, outcome $S$ .

NB: conservative solution for the Lipset data: $D\neg UL\neg IG+DULIG$ ; most parsimonious solution: $D\neg I+UG$ ; SUIN and necessary conditions, respectively, in bold font.

Table 3 displays the ESA most parsimonious solutions that are obtained under each of the four necessity claims that are potentially feasible based on QCA realist standards. As can be seen, none of these ESA solutions is identical to the conservative solution—i.e., there clearly is no CONSOL effect of ESA. ⁹ ESA makes sure, instead, that any sufficiency solution formula contains all the postulated SUIN or atomic necessary condition ¹⁰ and that no untenable assumptions on logical remainders are made.

On rare occasions and depending on the data at hand, in applied QCA it can, of course, happen that a QCA realist’s necessity claims only allow for the conservative solution in the analysis of sufficiency. Arguing that this is a design flaw of ESA is nothing else than beating the messenger. ESA just prevents researchers from making sufficiency claims that contradict their necessity claims—a laudable feature on which both QCA realists and idealists seem to agree.

4 Concluding Remarks—Idealists and Realists

The critique at ESA in Thiem (2017) is unconvincing and incoherent both for QCA realists and idealists. From a QCA realist’s perspective, Thiem’s approach to the analysis of necessary conditions and thus his critique at ESA is unconvincing because it simply ignores virtually all of the existing standards and practices. From QCA idealist’s perspective, his critique is incoherent for the simple reason that the very search for only necessary conditions has no place in that approach.

I have demonstrated that the tools at hand for researchers who locate themselves in the realist QCA camp are working fine. A meaningful claim of necessity must be empirically consistent, empirically non-trivial, and theoretical meaningful. Constraining computer-based search functions via additional empirical and theoretical criteria is nothing unusual and certainly not a form of bias as Thiem (2017) insinuates. This strategy usually does not yield the conservative solution, but if it does, then this is not a flaw of ESA but a feature of the data at hand.

QCA realists could, of course, ignore these and other intricacies of applied QCA and turn into idealists. Given the internal coherence and grounding in a regularity theory of causation, why has the idealist position not gained more traction so far? Why are most QCA researchers realists and perform separate analyses of necessity and sufficiency? Why are entire books written on necessary causes (usually single conditions) despite Thiem (2017) intoning that “[…] necessity is no relational property of an object that is conferred onto it by the causal interpretation of a researcher, but only by the algebraic laws that define the operation of necessity in conjunction with the data in hand.” (p. 480)? And why do QCA scholars often base their analysis on the intermediate or conservative solution when idealists argue—convincingly, I find, from the perspective of a regularity theory of causation—that only the most parsimonious solution allows for causal inference? Idealists seem to suggest that all this is explained by the majority of QCA researchers’ incapacity to grasp what the search target of QCA is.

I, instead, believe that QCA realists are fully aware that by allowing for less-than-perfect set relations, relying on solution formulas other than the most parsimonious, by separating the analyses of necessity and sufficiency etc. they pay the prize of not exclusively relying on pure formal logic nor a pure version of a regularity theory of causation. Realists are willing to pay this prize in order to make QCA applicable to, and useful for, real social science research. It is still unclear how the idealist position in its current form is fruitfully applied to social science theories and data. Claiming that QCA both is only good for, and always manages to, identifying the minimally necessary disjunctions of minimally sufficient conjunctions seemingly strikes QCA realists as impractical, for several reasons and with various strategies that have been developed in response.

First, most researchers know that the conditions under which a truth table analysis yields a causally interpretable minimally necessary disjunction of sufficient conjunctions are rarely ever met in applied QCA. QCA idealists would need to explain what researchers ought to do if and when consistency and coverage are (far) less than perfect. ¹¹ Second, QCA realists are probably also waiting for the idealists’ take on the glaring gap between social science theories on necessary conditions, which usually are about single conditions or simpler disjunctions, on the one hand, and the QCA idealist position that usually very complex expressions—disjunctions of conjunctions—are to be interpreted as the necessary conditions, on the other.

Third, most scholars with some experience in social science research do know that drawing causal inference based on observational data is a tall order—also, of course, in QCA. QCA idealists strongly emphasize that within a regularity theory of causation framework the redundant-free most parsimonious solution (with perfect consistency and coverage) is one necessary element for causal inference. ¹² Since, however, this is hardly ever found in real data, QCA realists adopt various strategies in response to noisy data. ¹³ For instance, some correctly point out that a QCA-based study can serve purposes other than drawing causal inference (e.g. Berg-Schlosser, De Meur, and Rihoux 2009). Others adhere to the goal of causal inference but point out that the logical minimization of a truth table based on observational data at the cross-case level can only be one element and is limited to establishing the presence of a causal effect. The (heavier) lifting will need to be done at the within-case level and is there to probe the presence of a causal mechanism. In the absence of the latter, no QCA solution formula, redundant-free or not, should count as causal. If the goal is to draw causal inference, it strikes me as fruitful to spell out the logic of how researchers should combine their QCA with follow-up case studies (e.g. Schneider and Rohlfing 2013, 2016). Since, however, QCA realists who pursue causal inference cannot fully rely on a regularity theory of causation as their common ground, they need to spell out which other theory, or theories, of causation could better undergird QCA as applied in the social sciences. ¹⁴

ESA, just like many other methodological proposals made by QCA realists, are developed based on the trivial insights that the data to be analysed comes short of the ideal scenarios of pure Boolean algebra and set theory and, more generally, that methodological tools need to align with social science theories and research questions. Unless QCA idealists come forth with a coherent set of proposals that is tested in real research rather than just talked about, applied QCA researchers seem better advised to adhere to, and to work on improving, the QCA realist standards and practices.

Supplementary material

For supplementary material accompanying this paper, please visit https://doi.org/10.1017/pan.2017.45.