8 - Potential pitfalls in the Standard Analysis procedure and suggestions for improvement

Summary

Beyond the Standard Analysis: expanding the types of counterfactuals

In section 6.4.3, we introduced the distinction between different types of assumptions, or counterfactual claims: simplifying assumptions, and easy and difficult counterfactuals. In the following, we further differentiate between types of counterfactuals. We will label them good counterfactuals and untenable counterfactuals, the latter consisting of implausible and incoherent counterfactuals.

Incoherent counterfactuals are defined as assumptions that contradict claims made about the same remainder at a different moment of the analytical process. This fallacy can happen in two ways. First, the researcher performs separate analyses for both Y and ~Y and includes the same remainder into both minimization procedures. By doing so, the researcher is effectively saying that the same logical remainder is sufficient for both the occurrence and the non-occurrence of an outcome. This type of assumption is already discussed in the literature under the label contradictory assumption (Yamasaki and Rihoux 2009 ). The second form of incoherent counterfactuals can occur when researchers make a claim of necessity but then also allow a logical remainder to be part of a sufficiency solution that contradicts that claim of necessity. To see how this happens, recall that if X is necessary for Y, then Y cannot occur in the presence of ~X. Formally: if X ←Y, then ~X → ~Y ( section 3.3.3 ). Hence, if a researcher claims that X is necessary for Y, then any logical remainder row that implies condition ~X cannot be used for a counterfactual claim following which it would produce outcome Y. This is, so far, a widely overlooked pitfall.