Hostname: page-component-cd9895bd7-q99xh Total loading time: 0 Render date: 2024-12-27T10:30:18.455Z Has data issue: false hasContentIssue false
  • English
  • Français

Nonresponse and Mode Effects in Self- and Interviewer-Administered Surveys

Published online by Cambridge University Press:  04 January 2017

Lonna Rae Atkeson ,
Alex N. Adams  and
R. Michael Alvarez
Lonna Rae Atkeson*
Affiliation:
Department of Political Science, University of New Mexico, MSC05-3070, 1 University of New Mexico, Albuquerque, NM 87131-0001
Alex N. Adams
Affiliation:
Department of Political Science, University of New Mexico, MSC05-3070, 1 University of New Mexico, Albuquerque, NM 87131-0001. email: alexnadams@yahoo.com
R. Michael Alvarez
Affiliation:
California Institute of Technology, 1200 E. California Blvd., Pasadena, CA 91125. email: rma@hss.caltech.edu
*
e-mail: atkeson@unm.edu (corresponding author)
Get access Rights & Permissions [Opens in a new window]

Abstract

We examine the quality of two probability-based polls, one interviewer administered (telephone) and one self-administered (Internet and mail mixed mode survey). The polls use the same sampling frame (registered voters) and the same questions. First, we examine the representativeness of both surveys using information known about the population, and although we find important differences between the two in terms of sampling and nonresponse bias, we also find that both surveys represent the underlying population despite low response rates. We also test for mode effects between surveys due to social desirability and how it influences nondifferentiation or satisficing. Using a variety of methods (t-tests, multivariate regression, and genetic propensity matching), we find evidence that the presence of an interviewer alters response patterns on ego-driven questions. The implications of our work are important, due to the increasing popularity of mixed mode surveys. Researchers need to be methodologically sensitive to these differences when analyzing surveys that allow for different response modes.

Type
Symposium on Advances in Survey Methodology
Information
Political Analysis , Volume 22 , Issue 3 , Summer 2014 , pp. 304 - 320
Copyright
Copyright © The Author 2014. Published by Oxford University Press on behalf of the Society for Political Methodology 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Authors' note: Supplementary materials for this article are available on the Political Analysis Web site.

References

Adams, Alex, Atkeson, Lonna, and Karp, Jeffrey. 2013. Measurement error in online survey panels: Panel conditioning and data quality. Presented at the American Panel Survey (TAPS) Workshop, Washington University, St. Louis, November 8–9.Google Scholar
American Association for Public Opinion Research. 2010. AAPOR report on online panels. http://www.aapor.org/AAPOR_Releases_Report_on_Online_Survey_Panels/2263.htm (accessed January 10, 2014).Google Scholar
The American Association for Public Opinion Research. 2011. Standard definitions: Final dispositions of case codes and outcome rates for surveys, 7th edition. AAPOR. http://aapor.org/Content/NavigationMenu/AboutAAPOR/StandardsampEthics/StandardDefinitions/StandardDefinitions2011.pdf (accessed January 10, 2014).Google Scholar
Alexander, C. H. Jr., and Wetrogan, S. 2000. Integrating the American Community Survey and the Intercensal Demographic Estimates Program. Proceedings of the American Statistical Association at www.amstat.org/sections/srms/proceedings/papers/2000_046.pdf (accessed January 10, 2014).Google Scholar
Aquilino, W. 1994. Interview mode effects in surveys of drug and alcohol use. Public Opinion Quarterly 58: 210-40.Google Scholar
Atkeson, Lonna Rae, Adams, Alex N., and Michael Alvarez, R. 2013. Replication data for: Nonresponse and mode effects in self- and interviewer-administered surveys. http://dx.doi.org/10.7910/DVN/23773 UNF:5:QH1dvPfRoJ+lbQkO32oeww== IQSS Dataverse Network [Distributor] V1 [Version].Google Scholar
Atkeson, Lonna Rae, and Adams, Alex N. 2010. Mixed mode (Internet and Mail) probability samples and survey representativeness: The case of New Mexico 2008. San Francisco, CA: Presented at the Western Political Science Association, April 1–4.Google Scholar
Atkeson, Lonna Rae, Adams, Alex N., Bryant, Lisa A., Zilberman, Luciana, and Saunders, Kyle L. 2011. Considering mixed mode surveys for questions in political behavior: Using the Internet and mail to get quality data at reasonable costs. Political Behavior 33 (1): 161-78.Google Scholar
Atkeson, Lonna Rae, and Tafoya, Lori. 2008. Surveying political activists: An examination of the effectiveness of a mixed- mode (Internet and mail) survey design. Journal of Elections Public Opinion and Parties 18 (4): 367-86.Google Scholar
Atkeson, Lonna Rae, Michael Alvarez, R., Adams, Alex N., and Bryant, Lisa. 2011. The 2010 New Mexico election administration report. Typescript, University of New Mexico. http://polisci.unm.edu/common/documents/c-sved/papers/nm-2010-general-election.pdf (accessed January 10, 2014).Google Scholar
Atkeson, Lonna Rae, Bryant, Lisa A., and Adams, Alex N. 2013. The 2012 Bernalillo County election administration report. Typescript, University of New Mexico. http://polisci.unm.edu/common/documents/c-sved/papers/BernCoFinalReport.pdf (accessed January 10, 2014).Google Scholar
Atkeson, Lonna Rae, Michael Alvarez, R., and Hall, Thad E. 2010. Assessing electoral performance in New Mexico using an eco-system approach: New Mexico 2008. Typescript, University of New Mexico. http://polisci.unm.edu/common/documents/c-sved/papers/nm08pew.pdf (accessed January 10, 2014).Google Scholar
Balden, W. 2004. Multi-mode data collection: Benefits and downsides. Paper presented at the 2004 Conference of the Great Lakes. Cancun, Mexico: Chapter of the Marketing Research Association.Google Scholar
Belli, Robert F., Traugott, Michael W., and Beckmann, Matthew N. 2001. What leads to voting overreports? Contrasts of overreporters and admitted nonvoters in the American National Election Studies. Journal of Official Statistics 17 (4): 479-98.Google Scholar
Blair, Graeme, and Imai, Kosuke. 2012. Statistical analysis of list experiments. Political Analysis 20: 47–77.CrossRefGoogle Scholar
Blumberg, Stephen J., and Luke, Julian V. 2009. Wireless substitution: Early release of estimates from the National Health Interview Survey. July-December 2008. http://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless200905.htm.Google Scholar
Blumberg, Stephen J., and Luke, Julian V. 2013. Wireless substitution: Early release of estimates from the National Health Interview Survey. July-December 2012. http://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless201306.pdf.Google Scholar
Burden, Barry C. 2000. Voter turnout and the National Election Studies. Political Analysis 8 (4): 389-98.CrossRefGoogle Scholar
Burden, Barry C. 2003. Internal and external effects on the accuracy of NES turnout: Reply. Political Analysis 11 (2): 193-95.Google Scholar
Campbell, Donald T. 1950. The indirect assessment of social attitudes. Psychological Bulletin 47):15–38.Google Scholar
Cannell, C. F., Miller, P. V., Oksenberg, and L. 1981. Research on interviewing techniques. In Sociological Methodology, ed. Leinhardt, S., 389–437. San Francisco, CA: Jossey-Bass.Google Scholar
Carley-Baxter, Lisa R., Peytchev, Andy, and Black, Michele C. 2010. Comparison of cell phone and landline surveys: A design perspective. Field Methods 22 (1): 3–15.Google Scholar
Chang, LinChiat, and Krosnick, Jon. 2009. National surveys via RDD telephone interviewing versus the Internet: Comparing sample representativeness and response quality. Public Opinion Quarterly 73 (4): 641-78.Google Scholar
Chang, LinChiat, and Krosnick, Jon. 2010. Comparing oral interviewing with self-administered computerized questionnaires: An experiment. Public Opinion Quarterly 74 (1): 154-67.Google Scholar
Christian Leah, M., and Dillman, Don A. 2004. The influence of graphical and symbolic language manipulations on responses to self-administered questions. Public Opinion Quarterly 68 (1): 57–80.Google Scholar
Curtin, Richard, Presser, Stanley, and Singer, Eleanor. 2000. The effects of response rate changes on the index of consumer sentiment. Public Opinion Quarterly 64: 413-28.Google Scholar
Davis, James Allan, Smith, Tom, and Marsden, Peter V. 2009. General social surveys, 1972–2008: Cumulative codebook. Chicago: National Opinion Research Center.Google Scholar
Day, N. A., Dunt, D. R., Day, and S. 1995. Maximizing response to surveys in health program evaluation at minimum cost using multiple methods. Evaluation Review 19 (4): 436-50.CrossRefGoogle Scholar
de Leeuw, Edith. 2005. To mix or not to mix: Data collection modes in surveys. Journal of Official Statistics 21: 233-55.Google Scholar
de Leeuw, E., Hox, J., Korendijk, E., Lensvelt-Mulders, G., Callegaro, and M. 2004. The influence of advance letters on response in telephone surveys: A meta-analysis. Maastricht: Paper presented at the 15 International Workshop on Household Survey Nonresponse.Google Scholar
de Leeuw, Edith D., and de Heer, W. 2002. Trends in household survey nonresponse: A longitudinal and international comparison. In Survey nonresponse, eds. Groves, Robert M., Dillman, Don A., Eltinge, John L., and Little, Roderick J. A. New York: Wiley.Google Scholar
de Leeuw, Edith, and Van der Zouwen, J. 1988. Data quality in telephone and face-to-face surveys: A comparative meta- analysis. In Telephone survey methodology, eds. Groves, R., Bimer, P. P., Lyberg, L., Massey, I. T., Nicholls, W. L., Waksberg, and J. New York: John Wiley & Sons.Google Scholar
DeBell, Mathew, Wilson, Catherine, Segura, Gary, Jackman, Simon, and Hutchings, Vincent. 2011. Methodology report and user's guide for the ANES 2010–2012 Evaluations of Government and Society Study. Palo Alto, CA, and Ann Arbor: Stanford University and University of Michigan.Google Scholar
Diamond, Alexis, and Sekhon, Jasjeet S. 2012. Genetic matching for estimating causal effects: A general multivariate matching method for achieving balance in observational studies. Review of Economics and Statistics. http://sekhon.berkeley.edu/papers/GenMatch.pdf.Google Scholar
Dillman, Don A. 2000. Mail and Internet surveys: The tailored design method. 2nd ed. New York: Wiley.Google Scholar
Dillman, Don A., Sangster, R., Tarnai, John, and Rockwood, T. 1996. In Current issues in survey research: New directions for evaluation, #70. Chapter 4, eds. Braverman, Mark T., and Kay Slater, Jana, 45–62. San Francisco, CA: Jossey-Bass.Google Scholar
Dillman, Don A., and Malani Christian, Leah. 2005. Survey mode as a source of instability in responses across surveys. Field Methods 17 (1): 30–52.Google Scholar
Dillman, Don A., Smyth, Jolene, and Melani Christian, Leah. 2009. Internet, mail, and mixed-mode surveys: The tailored design method. New York: Wiley.Google Scholar
Fowler, Floyd Jackson. Jr, Roman, Anthony M., and Xiao Di, Zhu. 1998. Mode effects in a survey of Medicare prostate surgery patients. Public Opinion Quarterly 62 (1): 29–46.Google Scholar
Fricker, S., Galesic, M., Tourangeau, R., Yan, and T. 2005. An experimental comparison of web and telephone surveys. Public Opinion Quarterly 69 (3): 370-92.Google Scholar
Gingerich, D. W. 2010. Understanding off-the-books politics: Conducting inference on the determinants of sensitive behavior with randomized response surveys. Political Analysis 18: 349-80.Google Scholar
Groves, Robert M. 1989. Survey errors and survey costs. New York: John Wiley and Sons.Google Scholar
Groves, R. M., Kahn, and R. L. 1979. Surveys by telephone: A national comparison with personal interviews. New York: Academic Press.Google Scholar
Groves, R. M., Peytcheva, and E. 2008. The impact of nonresponse rates on nonresponse bias. Public Opinion Quarterly 72: 1–23.Google Scholar
Groves, Robert M., Presser, Stanley, and Dipko, Sarah. 2004. The role of topic interest in survey participation decisions. Public Opinion Quarterly 68 (1): 2–31.Google Scholar
Groves, Robert, Singer, Eleanor, and Corning, Amy. 2000. Leverage-salience theory of survey participation. Public Opinion Quarterly 64: 299–308.Google Scholar
Ho, Daniel, Imai, Kosuke, King, Gary, and Stuart, Elizabeth. 2007a. Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis 15 (3): 199–236.Google Scholar
Ho, Daniel, Imai, Kosuke, King, Gary, and Stuart, Elizabeth. 2007b. Matchit: Nonparametric preprocessing for parametric causal inference. Journal of Statistical Software. http://gking.harvard.edu/matchit/.Google Scholar
Holbrook, A. L., Green, M. C., Krosnick, and J. A. 2003. Telephone versus face-to-face interviewing of national probability samples with long questionnaires. Public Opinion Quarterly 67: 79–125.Google Scholar
Holbrook, Allyson L., and Krosnick, Jon A. 2010. Social desirability bias in voter turnout reports: Tests using the item count technique. Public Opinion Quarterly 74 (1): 37–67.Google Scholar
Hox, J. J., and de Leeuw, E. D. 1994. A comparison of nonresponse in mail, telephone, and face-to-face surveys: Applying multilevel models to meta-analysis. Quality and Quantity 28: 329-44.Google Scholar
Iannacchione, Vincent. 2011. The changing role of address-based sampling in survey research. Public Opinion Quarterly 75 (3): 556-75.Google Scholar
Keeter, Scott, Miller, Carolyn, Kohut, Andrew, Groves, Robert M., and Presser, Stanley. 2000. Consequences of reducing non-response in a national telephone survey. Public Opinion Quarterly 64: 125-48.Google Scholar
Kreuter, F., Presser, S., Tourangeau, and R. 2008. Social desirability bias in CATI, IVR, and web surveys: The effects of mode and question sensitivity. Public Opinion Quarterly 72: 847-65.Google Scholar
Krosnick, J. A. 1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology 5: 213-36.Google Scholar
Krosnick, J. A. 1999. Maximizing questionnaire quality. In Measures of political attitudes, eds. Robinson, J. P., Shaver, P. R., Wrightsman, and L. S. New York: Academic Press.Google Scholar
London, K., Williams, and L. 1990. A comparison of abortion underreporting in an in-person interview and self-administered questionnaire. Toronto: Paper presented at the Annual Meeting of the Population Association of America.Google Scholar
Lupia, Arthur, Krosnick, Jon, Luevano, Pat, DeBell, Matthew, and Donakowski, Darrell. 2009. Users Guide to the ANES 2008 Time Series Study. Ann Arbor, MI, and Palo Alto, CA: University of Michigan and Stanford University. http://www.electionstudies.org/studypages/2008prepost/2008prepost_UsersGuide.pdf.Google Scholar
Martinez, Michael D. 2003. Comment on voter turnout and the National Election Studies. Political Analysis 11 (2): 187-92.Google Scholar
McDonald, Michael P. 2003. On the overreport bias of the National Election Study turnout rate. Political Analysis 11 (2): 180-86.Google Scholar
McDonald, Michael P. 2007. The true electorate: A cross validation of voter registration files and election survey demographics. Public Opinion Quarterly 71 (4): 588–602.Google Scholar
Mebane, Walter R. Jr, and Sekhon, Jasjeet S. 1998. GENetic Optimization Using Derivatives (GENOUD). Software package. http://sekhon/berkeley.edu/rgenoud/.Google Scholar
Mensch, Barbara S., and Kandel, Denise B. 1988. Underreporting of substance use in a National Longitudinal Youth Cohort. Public Opinion Quarterly 52 (Spring): 100-24.Google Scholar
Merkle, Daniel M., and Edelman, Murray. 2002. Non-response in exit polls: A comprehensive analysis. In Survey non- response, eds. Groves, Robert M., Dillman, Don A., Eltinge, John L., and Little, Roderick J. A., 243-58. New York: John Wiley & Sons Inc. Google Scholar
Mulligan, Kenneth, Krosnick, Jon A., Smith, Wendy, Green, Melanie, and Bizer, George. 2001. Nondifferentiation on attitude rating scales: A test of survey satisficing theory. Unpublished manuscript.Google Scholar
O'Muircheartaigh, Colm, English, Ned, Latterner, Michael, Eckman, Stephanie, and Dekker, Katie. 2009. Modeling the need for traditional vs. commercially available address listings for in-person surveys: Results from a National Validation of Addresses JSM. Proceedings of the American Statistical Association, Section on Survey Research Methods 6193–202.Google Scholar
Peterson, Robert A., and Kerin, Roger A. 1981. The quality of self-report data: Review and synthesis. In Review of marketing, eds. Enis, Ben M. and Roering, Kenneth J., 5–20. Chicago: American Marketing Association.Google Scholar
Pew Research Center for the People and the Press. 2004. Polls face growing resistance, but still representative: Survey experiment shows. http://people-press.org/reportsdisplay.php3?ReportID=211.Google Scholar
Presser, Stanley, Traugott, Michael W., and Traugott, Santa. 1990. Vote “over” reporting in surveys: The records or the respondents. Technical Report No. 39. Ann Arbor, MI: National Election Studies.Google Scholar
Rivers, Douglas. n.d. Sample matching: Representative sampling from Internet polling. Polimetrix.Google Scholar
Rosen, R., Gomes, and T. 2004. Converting CES reporters from TDE to web data collection. Toronto, Canada: Paper presented at the Joint Statistical Meetings.Google Scholar
Schuman, Howard, and Presser, Stanley. 1981. Questions and answers in attitude surveys: Experiments on question form, wording and context. New York: Academic Press.Google Scholar
Schuman, Howard. 1992. Context effects: State of the past/state of the art. In Context effects in social and psychological research, eds. Schwarz, N. and Sudman, S., 5–20. New York: Springer.Google Scholar
Schwarz, N. 1996. Cognition and communication judgmental biases, research methods, and the logic of conversation. Mahwah, NJ: Lawrence Erlbaum.Google Scholar
Schwarz, N., Strack, and E. 1985. Cognitive and affective processes in judgments of subjective well-being: A preliminary model. In Economic psychology, eds. Brandstatter, H. and Kirehler, E., 439-47. Linz: Austria: R. Trauner.Google Scholar
Sekhon, Jasjeet Singh, and Mebane, Walter R. Jr. 1998. Genetic optimization using derivatives: Theory and application to nonlinear models. Political Analysis 7: 189–203.Google Scholar
Sekhon, Jasjeet S. 2011. Multivariate and propensity score matching software with automated balance optimization: The matching package for R. Journal of Statistical Software 42 (7): 1–52.Google Scholar
Shettle, Carolyn, and Mooney, Geraldine. 1999. Monetary incentives in government surveys. Journal of Official Statistics 15: 231-50.Google Scholar
Smyth, Jolene D., Dillman, Don, Malani Christian, Leah, and Stern, Michael J. 2006. Effects of using visual design principles to group response options in web surveys. International Journal of Internet Science 1: 6–16.Google Scholar
Stoop, I. A. L. 2005. The hunt for the last respondent. The Hague: Social and Cultural Planning Office (SCP).Google Scholar
Seymour, Sudman, Bradburn, Norman M., and Schwarz, Norbert. 1996. Thinking about answers. San Francisco, CA: Josey-Bass.Google Scholar
Tourangeau, Roger, Couper, Mick, and Conrad, Frederick. 2004. Spacing, position and order: Interpretive heuristics for visual features of survey questions. Public Opinion Quarterly 68 (3): 368-93.CrossRefGoogle Scholar
Tourangeau, R., Rasinski, and K. A. 1988. Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin 103: 299–314.Google Scholar
Tourangeau, Roger, and Yan, Ting. 2007. Sensitive questions in surveys. Psychological Bulletin 133: 859-83.Google Scholar
Traugott, Santa. 1989. Validating self-reported vote: 1964–1988. ANES Technical Report Series, No. nes010152. Ann Arbor, MI: National Election Studies.Google Scholar
U.S Census Bureau. American community survey 2006–2008 three-year estimates. Data profile: New Mexico: Bernalillo County: Hispanics (All).Google Scholar
Visser, Penny S., Krosnick, Jon A., Marquette, Jesse, and Curtin, Michael. 1996. Mail surveys for election forecasting? An evaluation of the Columbus Dispatch Poll. Public Opinion Quarterly 60 (2): 181–227.Google Scholar
Zickhur, K., Smith, and A. 2012. Digital differences. Washington, DC: Pew Internet and American Life Project. http://pewinternet.org/Reports/2012/Digital-differences.aspx (accessed August 7, 2012).Google Scholar
Supplementary material: PDF

Atkeson et al. supplementary material

Tables S1-S4

Download Atkeson et al. supplementary material(PDF)
PDF 99.6 KB