Hostname: page-component-745bb68f8f-kw2vx Total loading time: 0 Render date: 2025-01-14T07:10:11.189Z Has data issue: false hasContentIssue false
  • English
  • Français

The Logic of the Survey Experiment Reexamined

Published online by Cambridge University Press:  04 January 2017

Brian J. Gaines ,
James H. Kuklinski  and
Paul J. Quirk
Brian J. Gaines
Affiliation:
Department of Political Science and Institute of Government and Public Affairs, University of Illinois at Urbana-Champaign, 702 South Wright Street, Urbana, IL 61801. e-mail: bjgaines@uiuc.edu (corresponding author)
James H. Kuklinski
Affiliation:
Department of Political Science and Institute of Government and Public Affairs, University of Illinois at Urbana-Champaign, 702 South Wright Street, Urbana, IL 61801. e-mail: kuklinsk@ad.uiuc.edu
Paul J. Quirk
Affiliation:
Department of Political Science, University of British Columbia, 2329 West Mall, Vancouver, British Columbia, Canada, V6T 1Z4. e-mail: quirk@politics.ubc.ca
Get access Rights & Permissions [Opens in a new window]

Abstract

Scholars of political behavior increasingly embed experimental designs in opinion surveys by randomly assigning respondents alternative versions of questionnaire items. Such experiments have major advantages: they are simple to implement and they dodge some of the difficulties of making inferences from conventional survey data. But survey experiments are no panacea. We identify problems of inference associated with typical uses of survey experiments in political science and highlight a range of difficulties, some of which have straightforward solutions within the survey-experimental approach and some of which can be dealt with only by exercising greater caution in interpreting findings and bringing to bear alternative strategies of research.

Type
Research Article
Information
Political Analysis , Volume 15 , Issue 1 , Winter 2007 , pp. 1 - 20
Copyright
Copyright © The Author 2006. Published by Oxford University Press on behalf of the Society for Political Methodology 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Achen, Christopher H. 1986. The statistical analysis of quasi-experiments. Berkeley, CA: University of California Press.CrossRefGoogle Scholar
Ansolabehere, Stephen, and Iyengar, Shanto. 1995. Going negative: How attack ads shrink and polarize the electorate. New York: Free Press.Google Scholar
Brady, Henry E., and Collier, David, eds. 2000. Rethinking social inquiry: Diverse tools, shared standards. Lanham, MD: Rowman and Littlefield.Google Scholar
Clarke, Harold D., Kornberg, Allan, McIntyre, Chris, Bauer-Kaase, Petra, and Kaase, Max. 1999. The effect of economic priorities on the measurement of value change: New experimental evidence. American Political Science Review 93: 637–47.CrossRefGoogle Scholar
Davis, Darren W., and Silver, Brian D. 2003. Stereotype threat and race of interviewer effects in a survey on political knowledge. American Journal of Political Science 47: 33–45.Google Scholar
Druckman, James N. 2001. The implications of framing effects for citizen competence. Political Behavior 23: 225–56.CrossRefGoogle Scholar
Druckman, James N., and Lupia, Arthur. 2006. Mind, will, and choice. In The Oxford handbook on contextual political analysis, ed. Tilly, Charles and Goodwin, Robert E., 97–113. Oxford: Oxford University Press.Google Scholar
Druckman, James N., and Nelson, Kjersten R. 2003. Framing and deliberation: How citizens' conversations limit elite influence. American Journal of Political Science 47: 729–45.CrossRefGoogle Scholar
Fearon, James D. 1991. Counterfactuals and hypothesis testing in political science. World Politics 43: 169–95.CrossRefGoogle Scholar
Gibson, James. 1998. A sober second thought: An experiment in persuading Russians to tolerate. American Journal of Political Science 42: 819–50.CrossRefGoogle Scholar
Gilens, Martin. 1999. Why Americans hate welfare: Race, media, and the politics of anti-poverty reform. Chicago, IL: University of Chicago Press.Google Scholar
Gilens, Martin. 2001. Political ignorance and collective policy preferences. American Political Science Review 95: 379–98.Google Scholar
Grant, J. Tobin, and Rudolph, Thomas J. 2003. Value conflict, group affect, and the issue of campaign finance. American Journal of Political Science 47: 453–69.CrossRefGoogle Scholar
Heckman, James J., Ichimura, Hidehiko, and Todd, Petra E. 1997. Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme. The Review of Economic Studies 64: 605–54.Google Scholar
Heckman, James J., Ichimura, Hidehiko, and Todd, Petra E. 1998. Matching as an econometric evaluation estimator. The Review of Economic Studies 65: 261–94.CrossRefGoogle Scholar
Heckman, James, Ichimura, Hidehiko, Smith, Jeffrey, and Todd, Petra. 1998. Characterizing selection bias using experimental data. Econometrica 66: 1017–98.CrossRefGoogle Scholar
Hovland, Carl I., and Weiss, Walter. 1951. The influence of source credibility on communication effectiveness. Public Opinion Quarterly 15: 635–50.CrossRefGoogle Scholar
Hurwitz, Jon, and Peffley, Mark. 1997. Public perception of race and crime: The role of racial stereotypes. American Journal of Political Science 41: 375–401.CrossRefGoogle Scholar
Hyman, Herbert H., and Sheatsley, Paul B. 1950. The current status of American public opinion. In The teaching of contemporary affairs, ed. Payne, J. C., 11–34. Washington, DC: National Council for the Social Studies.Google Scholar
Inglehart, Ronald. 1990. Culture shift in advanced industrial society. Princeton, NJ: Princeton University Press.Google Scholar
Kinder, Donald R., and Sanders, Lynn M. 1996. Divided by color: Racial politics and democratic ideals. Chicago, IL: University of Chicago Press.Google Scholar
King, Gary, Keohane, Robert O., and Verba, Sidney. 1994. Designing social inquiry: Scientific inference in qualitative research. Princeton, NJ: Princeton University Press.CrossRefGoogle Scholar
Kuklinski, James H., Cobb, Michael D., and Gilens, Martin. 1997. Racial attitudes and the “New South.” Journal of Politics 59: 323–49.Google Scholar
Kuklinski, James H., Riggle, Ellen, Ottati, Victor, Schwarz, Norbert, and Wyer, Robert S. Jr. 1991. The cognitive and affective bases of political tolerance judgments. American Journal of Political Science 35: 1–27.CrossRefGoogle Scholar
Lacy, Dean. 2001. A theory of nonseparable preferences in survey responses. American Journal of Political Science 45: 239–58.Google Scholar
LaLonde, Robert J. 1986. Evaluating the econometric evaluations of training programs with experimental data. American Economic Review 76: 604–20.Google Scholar
Lieberson, Stanley. 1987. Making it count: The improvement of social research and theory. Berkeley, CA: University of California Press.Google Scholar
Lodge, Milton, Steenbergen, Marco R., and Brau, Shawn. 1995. The responsive voter: Campaign information and the dynamics of candidate evaluation. American Political Science Review 89: 309–26.Google Scholar
Lupia, Arthur, and McCubbins, Mathew D. 1998. The democratic dilemma: Can citizens learn what they need to know? Cambridge: Cambridge University Press.Google Scholar
Luskin, Robert C., Fishkin, James S., and Jowell, Roger. 2002. Considered opinions: Deliberative polling in Britain. British Journal of Political Science 32: 455–87.Google Scholar
Mondak, Jeffery J. 1993. Source cues and policy approval: The cognitive dynamics of public support for the Reagan agenda. American Journal of Political Science 37: 186–212.Google Scholar
Mutz, Diana C. 1998. Impersonal influence: How perceptions of mass collectives affect political attitudes. New York: Cambridge University Press.Google Scholar
Mutz, Diana C., and Reeves, Byron. 2005. The new videomalaise: Effects of televised civility on political trust. American Political Science Review 99: 1–15.CrossRefGoogle Scholar
Nelson, Thomas E., Clawson, Rosalee A., and Oxley, Zoe M. 1997. Media framing of a civil liberties conflict and its effect on tolerance. American Political Science Review 91: 567–83.Google Scholar
Peffley, Mark, Hurwitz, Jon, and Sniderman, Paul M. 1997. Racial stereotypes and whites' political views of blacks in the context of welfare and crime. American Journal of Political Science 41: 30–60.Google Scholar
Rosenbaum, Paul R., and Rubin, Donald B. 1983. The central role of the propensity score in observational studies for causal effects. Biometrika 70: 41–55.CrossRefGoogle Scholar
Schuman, Howard, and Bobo, Lawrence. 1988. Survey-based experiments on white racial attitudes toward residential segregation. American Journal of Sociology 94: 273–99.Google Scholar
Smith, Alastair. 1999. Testing theories of strategic choice: The example of crisis escalation. American Journal of Political Science 43: 1254–83.Google Scholar
Sniderman, Paul M. 1996. Innovations in experimental design in attitude surveys. Annual Review of Sociology 22: 377–99.CrossRefGoogle Scholar
Sniderman, Paul M., Brody, Richard A., and Tetlock, Philip E. 1991. Reasoning and choice: Explorations in political psychology. Cambridge: Cambridge University Press.Google Scholar
Sniderman, Paul M., and Carmines, Edward G. 1997. Reaching beyond race. Cambridge, MA: Harvard University Press.CrossRefGoogle Scholar
Sniderman, Paul M., and Piazza, Thomas. 1993. The scar of race. Cambridge, MA: Belknap Press.CrossRefGoogle Scholar
Sniderman, Paul M., and Theriault, Sean. 2004. The dynamics of political argument and the logic of issue framing. In Studies in public opinion: Attitudes, nonattitudes, measurement error, and change, ed. Saris, Willem E. and Sniderman, Paul M., 133–65. Princeton, NJ: Princeton University Press.Google Scholar
Sniderman, Paul M., Hagendoorn, Louk, and Prior, Markus. 2004. Predispositional factors and situational triggers: Exclusionary reactions to immigrant minorities. American Political Science Review 98: 35–50.Google Scholar
Taber, Charles S., and Lodge, Milton. 2006. Motivated skepticism in the evaluation of political beliefs. American Journal of Political Science 50: 755–69.Google Scholar
Zaller, John. 1992. The nature and origins of mass opinion. New York: Cambridge University Press.CrossRefGoogle Scholar