5 - Combining Case Studies and Matching

Summary

Case-study researchers have long used a certain interpretation of Mill's method of difference as a rationale for drawing causal conclusions from paired comparisons (e.g. Skocpol 1979: 36–39). The underlying argument can be translated into the potential-outcomes framework in a way that makes it more precise and simultaneously reveals why causal inferences based on paired comparisons are fundamentally unworkable in the social sciences.

Suppose a scholar is particularly interested in understanding the relationship between a treatment, D, and an outcome, Y, in case i. What must be true of a second case, j, in order for that case to allow causal inference via paired comparison with case i? Obviously, it must have the opposite score on D; for present purposes, let us assume that case i receives the treatment condition and case j the control.

More important and challenging requirements involve the counterfactuals. The second case in a paired comparison is intended to help reveal the unobserved potential outcome for the first case, and vice versa. That is to say, the observed value of Y_j,c must reveal useful information about the unobserved quantity Y_i,c, while Y_i,t must be informative about the counterfactual Y_j,t .

How can the value of Y_j,c be inferentially connected to Y_i,c? The most common rationale for making such an inference is to claim that the two values have to be identical because the two cases are causally indistinguishable: they have identical values on all variables causally connected with the outcome other than the treatment, and are identical with respect to any background traits that relate to the causal efficacy of the treatment.

An example may help illustrate the strictness of this idea of causally indistinguishable cases. Consider a chemistry experiment in which a large bottle of a compound is split into two smaller beakers, both of which were thoroughly cleaned just before the start of the experiment. If no contamination has taken place, then the two new beakers should be causally indistinguishable: there is no reason to suppose that they will behave differently unless a treatment is applied to one but not the other (see Holland 1986: 948). In such a context, a causal inference might plausibly be drawn by subjecting one sample to the treatment of interest and comparing its behavior to that of the control sample, because there is reason to believe that the two samples share identical potential outcomes.