Hostname: page-component-cd9895bd7-dzt6s Total loading time: 0 Render date: 2024-12-27T13:26:33.416Z Has data issue: false hasContentIssue false
  • English
  • Français

The Dangers of Extreme Counterfactuals

Published online by Cambridge University Press:  04 January 2017

Gary King  and
Langche Zeng
Gary King
Affiliation:
Department of Government and Institute for Quantitative Social Science, Harvard University, 1737 Cambridge Street, Cambridge MA 02138. e-mail: king@harvard.edu
Langche Zeng
Affiliation:
Department of Political Science, University of California, San Diego, 9500 Gilman Drive, La Jolla, CA 92093-0521. e-mail: zeng@ucsd.edu
Rights & Permissions [Opens in a new window]

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

We address the problem that occurs when inferences about counterfactuals—predictions, “what-if” questions, and causal effects—are attempted far from the available data. The danger of these extreme counterfactuals is that substantive conclusions drawn from statistical models that fit the data well turn out to be based largely on speculation hidden in convenient modeling assumptions that few would be willing to defend. Yet existing statistical strategies provide few reliable means of identifying extreme counterfactuals. We offer a proof that inferences farther from the data allow more model dependence and then develop easy-to-apply methods to evaluate how model dependent our answers would be to specified counterfactuals. These methods require neither sensitivity testing over specified classes of models nor evaluating any specific modeling assumptions. If an analysis fails the simple tests we offer, then we know that substantive results are sensitive to at least some modeling choices that are not based on empirical evidence. Free software that accompanies this article implements all the methods developed.

Type
Research Article
Information
Political Analysis , Volume 14 , Issue 2 , Spring 2006 , pp. 131 - 159
Copyright
Copyright © The Author 2005. Published by Oxford University Press on behalf of the Society for Political Methodology 

References

Bishop, Christopher M. 1995. Neural Networks for Pattern Recognition. Oxford: Oxford University Press.Google Scholar
Cuadras, C. M., and Fortiana, J. 1995. “A Continuous Metric Scaling Solution for a Random Variable.” Journal of Multivariate Analysis 52: 1–14.Google Scholar
Cuadras, C. M., Fortiana, J., and Oliva, F. 1997. “The Proximity of an Individual to a Population with Applications to Discriminant Analysis.” Journal of Classification 14: 117–136.CrossRefGoogle Scholar
de Berg, Mark, van Krevald, Marc, Overmars, Mark, and Schwarzkopf, Otfried. 1998. Computational Geometry: Algorithms and Applications, 2nd rev. ed. New York: Springer.Google Scholar
Esty, Daniel C., Goldstone, Jack, Gurr, Ted Robert, Harff, Barbara, Surko, Pamela T., Unger, Alan N., and Chen, Robert S. 1998. The State Failure Task Force Report: Phase II Findings. McLean, VA: Science Applications International Corporation.Google Scholar
Frangakis, Constantine E., and Rubin, Donald. 2002. “Principal Stratification in Causal Inference.” Biometrics 58: 21–29.CrossRefGoogle ScholarPubMed
Gelman, Andrew, and King, Gary. 1994. “Party Competition and Media Messages in U.S. Presidential Election Campaigns.” In The Parties Respond: Changes in the American Party System, ed. Maisel, Sandy L. Boulder, CO: Westview, pp. 255–295. (Available from http://gking.harvard.edu/files/abs/partycomp-abs.shtml.)Google Scholar
Gower, J. C. 1966. “Some Distance Properties of Latent Root and Vector Methods Used in Multivariate Analysis.” Biometrika 53 (3/4): 325–388.CrossRefGoogle Scholar
Gower, J. C. 1971. “A General Coefficient of Similarity and Some of Its Properties.” Biometrics 27: 857–872.Google Scholar
Greenland, Sander, Pearl, Judea, and Robins, James M. 1999. “Causal Diagrams for Epidemiologic Research.” Epidemiology 10(1): 37–48.Google Scholar
Hastie, Trevor, Tibshirani, R., and Friedman, J. 2001. The Elements of Statistical Learning. New York: Springer Verlag.Google Scholar
Heckman, James, Ichimura, H., and Todd, P. 1998. “Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Program.” Review of Economic Studies 64: 605–654.Google Scholar
Heckman, James J., Ichimura, Hidehiko, Smith, Jeffrey, and Todd, Petra. 1998. “Characterizing Selection Bias Using Experimental Data.” Econometrika 66(5): 1017–1098.Google Scholar
Ho, Daniel, Imai, Kosuke, King, Gary, and Stuart, Elizabeth. 2005. “Matching as Nonparametric Preprocessing for Parametric Causal Inference.” http://gking.harvard.edu/files/matchp.pdf.Google Scholar
Hoeting, Jennifer A., Madigan, David, Raftery, Adrian E., and Volinsky, Chris T. 1999. “Bayesian Model Averaging: A Tutorial.” Statistical Science 14(4): 382–417.Google Scholar
Holland, Paul W. 1986. “Statistics and Causal Inference.” Journal of the American Statistical Association 81: 945–960.Google Scholar
Imai, Kosuke, and van Dyk, David A. 2004. “Causal Inference with General Treatment Regimes: Generalizing the Propensity Score.” Journal of the American Statistical Association 99(467): 854–866.Google Scholar
Imai, Kosuke, and King, Gary. 2004. “Did Illegal Overseas Absentee Ballots Decide the 2000 U.S. Presidential Election?” Perspectives on Politics 2(3): 537–549.CrossRefGoogle Scholar
Kallay, Michael. 1986. “Convex Hull Made Easy.” Information Processing Letters 22 (March): 161.CrossRefGoogle Scholar
King, Gary. 1991. “‘Truth’ Is Stranger than Prediction, More Questionable than Causal Inference.” American Journal of Political Science 35(4): 1047–1053. http://gking.harvard.edu/files/abs/truth-abs.shtml.Google Scholar
King, Gary, Keohane, Robert O., and Verba, Sidney. 1994. Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton, NJ: Princeton University Press.Google Scholar
King, Gary, Tomz, Michael, and Wittenberg, Jason. 2000. “Making the Most of Statistical Analyses: Improving Interpretation and Presentation.” American Journal of Political Science 44(2): 341–355. http://gking.harvard.edu/files/abs/making-abs.shtml.Google Scholar
King, Gary, and Zeng, Langche. 2002. “Improving Forecasts of State Failure.” World Politics 53(4): 623–658. http://gking.harvard.edu/files/abs/civil-abs.shtml.CrossRefGoogle Scholar
Klee, Victor. 1980. “On the Complexity of d-Dimensional Voronoi Diagrams.” Archive der Mathematik 34: 75–80.Google Scholar
Kuo, Yen-Hong. 2001. “Extrapolation of Association between Two Variables in Four General Medical Journals.” Presented at the Fourth International Congress on Peer Review in Biomedical Publication, Barcelona, Spain.Google Scholar
Lechner, Michael. 1999. Identification and Estimation of Causal Effects of Multiple Treatments under the Conditional Independence Assumptions.” IZA Discussion Papers no. 91, University St. Gallen.CrossRefGoogle Scholar
Madych, W. R., and Nelson, S. A. 1992. Bounds on Multivariate Polynomials and Exponential Error Estimates for Multiquadric Interpolation.” Journal of Approximation Theory 70: 94–114.Google Scholar
Manski, Charles F. 1995. Identification Problems in the Social Sciences. Cambridge, MA: Harvard University Press.Google Scholar
Meng, Xiao-Li, and Romero, Marin. 2003. “Discussion: Efficiency and Self-Efficiency.” International Statistical Review 71(3): 607–618.CrossRefGoogle Scholar
O'Rourke, Joseph. 1998. Computational Geometry in C. New York: Cambridge University Press.CrossRefGoogle Scholar
Pearl, Judea. 2000. Causality: Models, Reasoning, and Inference. Cambridge, UK: Cambridge University Press.Google Scholar
Robins, James M. 1999a. Marginal Structural Models versus Structural Nested Models as Tools for Causal Inference.” In Statistical Models in Epidemiology: The Environment and Clinical Trials, vol. 16, eds. Halloran, M. E. and Berry, D. New York: Springer-Verlag, pp. 95–134.Google Scholar
Robins, James M. 1999b. Association, Causation, and Marginal Structural Models.” Synthese 121: 151–179.Google Scholar
Rosenbaum, Paul. 1984. “The Consequences of Adjusting for a Concomitant Variable That Has Been Affected by the Treatment.” Journal of the Royal Statistical Society, A 147(5): 656–666.Google Scholar
Rosenbaum, Paul R., and Rubin, Donald B. 1983. The Central Role of the Propensity Score in Observational Studies for Causal Effects.” Biometrika 70: 41–55.Google Scholar
Rosenbaum, Paul R., and Rubin, Donald B. 1984. Reducing Bias in Observational Studies Using Subclassification on the Propensity Score.” Journal of the American Statistical Association 79: 515–524.CrossRefGoogle Scholar
Rubin, Donald B. 1974. Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies.” Journal of Educational Psychology 6: 688–701.CrossRefGoogle Scholar
Schaback, R. 1996. Approximation by Radia Basis Functions with Finitely Many Centers.” Constructive Approximation 12: 331–340.Google Scholar
Sisson, Scott A. 2005. Transdimensional Markov Chains: A Decade of Progress and Future Perspectives.” Journal of the American Statistical Association 100(471): 1077–1089.CrossRefGoogle Scholar
Stoll, Heather, King, Gary, and Zeng, Langche. 2005. WhatIf: Software for Evaluating Counterfactuals.” http://gking.harvard.edu/whatif/.CrossRefGoogle Scholar
Valentine, Frederick Albert. 1964. Convex Sets. New York: McGraw-Hill.Google Scholar
Winship, Christopher, and Morgan, Stephen L. 1999. The Estimation of Causal Effects from Observational Data.” American Review of Sociology 25: 659–707.CrossRefGoogle Scholar
Wu, Z., and Schaback, R. 1993. Local Error Estimates for Radial Basis Function Interpolation of Scattered Data.” Journal of Numerical Analysis 13: 13–27.Google Scholar