Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-cd9895bd7-q99xh Total loading time: 0 Render date: 2024-12-27T08:30:42.609Z Has data issue: false hasContentIssue false
Series:   Elements in Quantitative and Computational Methods for the Social Sciences

Target Estimation and Adjustment Weighting for Survey Nonresponse and Sampling Bias

Published online by Cambridge University Press:  29 September 2020

Devin Caughey ,
Adam J. Berinsky ,
Sara Chatfield ,
Erin Hartman ,
Eric Schickler  and
Jasjeet S. Sekhon
Devin Caughey
Affiliation:
Massachusetts Institute of Technology
Adam J. Berinsky
Affiliation:
Massachusetts Institute of Technology
Sara Chatfield
Affiliation:
University of Denver
Erin Hartman
Affiliation:
University of California, Los Angeles
Eric Schickler
Affiliation:
University of California, Berkeley
Jasjeet S. Sekhon
Affiliation:
University of California, Berkeley

Summary

We elaborate a general workflow of weighting-based survey inference, decomposing it into two main tasks. The first is the estimation of population targets from one or more sources of auxiliary information. The second is the construction of weights that calibrate the survey sample to the population targets. We emphasize that these tasks are predicated on models of the measurement, sampling, and nonresponse process whose assumptions cannot be fully tested. After describing this workflow in abstract terms, we then describe in detail how it can be applied to the analysis of historical and contemporary opinion polls. We also discuss extensions of the basic workflow, particularly inference for causal quantities and multilevel regression and poststratification.
Get access

Keywords

weightingcalibrationecological inferencenonresponsequota sampling
Type
Element
Information
Series: Elements in Quantitative and Computational Methods for the Social Sciences
Online ISBN: 9781108879217
Publisher: Cambridge University Press
Print publication: 22 October 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Andridge, Rebecca R., and Little, Roderick J. A.. 2011. “Proxy Pattern-Mixture Analysis for Survey Nonresponse.” Journal of Official Statistics 27 (2): 153–180.Google Scholar
Ansolabehere, Stephen, and Hersh, Eitan. 2012. “Validation: What Big Data Reveal About Survey Misreporting and the Real Electorate.” Political Analysis 20 (4): 437–459.Google Scholar
Ansolabehere, Stephen, and Rivers, Douglas. 2013. “Cooperative Survey Research.” Annual Review of Political Science 16: 307–329.Google Scholar
Aronow, Peter M., and Miller, Benjamin T.. 2019. Foundations of Agnostic Statistics. New York: Cambridge University Press.Google Scholar
Baum, Matthew A., and Kernell, Samuel. 2001. “Economic Class and Popular Support for Franklin Roosevelt in War and Peace.” Public Opinion Quarterly 65 (2): 198–229.Google Scholar
Berinsky, Adam J. 2006. “American Public Opinion in the 1930s and 1940s: The Analysis of Quota-Controlled Sample Survey Data.” Public Opinion Quarterly 70 (4): 499–529.Google Scholar
Berinsky, Adam J., Neff Powell, Eleanor, Schickler, Eric, and Brett Yohai, Ian. 2011. “Revisiting Public Opinion in the 1930s and 1940s.” PS: Political Science & Politics 44 (3): 515–520.Google Scholar
Berinsky, Adam J., and Eric, Schickler. 2011. The American Mass Public in the 1930s and 1940s [Computer file]. Individual surveys conducted by the Gallup Organization [producers], 1936–1945: Roper Center for Public Opinion Research, University of Connecticut [distributor].Google Scholar
Bethlehem, Jelke G. 1988. “Reduction of Nonresponse Bias Through Regression Estimation.” Journal of Official Statistics 4 (3): 251–260.Google Scholar
Bethlehem, Jelke, Cobben, Fannie, and Schouten, Barry. 2011. Handbook of Nonresponse in Household Surveys. Hoboken, NJ: Wiley.Google Scholar
Binder, D. A., and Theberge, A.. 1988. “Estimating the Variance of Raking- Ratio Estimators.” Canadian Journal of Statistics 16: 47–55.Google Scholar
Brick, J. Michael, and Montaquila, Jill M.. 2009. “Nonresponse and Weighting.” In Sample Surveys: Design, Methods and Applications, edited by Pfeffermann, Danny and Rao, C. R., vol. 29A, 163–185. Handbook of Statistics. Elsevier.Google Scholar
British Polling Council. 2016. “Performance of the Polls in the EU Referendum.” June 24. Accessed October 26, 2019. www.britishpollingcouncil.org/performance-of-the-polls-in-the-eu-referendum/.Google Scholar
Bunche, Ralph J. 1973. The Political Status of the Negro in the Age of FDR. Edited by Grantham, Dewey W.. Chicago: University of Chicago Press.Google Scholar
Caldeira, Gregory A. 1987. “Public Opinion and the U.S. Supreme Court: FDR’s Court-Packing Plan.” American Political Science Review 81 (4): 1139–1153.Google Scholar
Caughey, Devin, and Hartman, Erin. 2017. “Target Selection as Variable Selection: Using the Lasso to Select Auxiliary Vectors for the Construction of Survey Weights.” Paper presented at the Annual Meeting of the Society for Political Methodology, University of Wisconsin-Madison, Madison, WI, July 13. https://ssrn.com/abstract=3494436.Google Scholar
Caughey, Devin, and Wang, Mallory. 2014. “Bayesian Population Interpolation and Lasso-Based Target Selection in Survey Weighting.” Paper presented at the Annual Meeting of the Society for Political Methodology, University of Georgia, Athens, GA, July 24. https://ssrn.com/abstract=3494430.Google Scholar
Caughey, Devin, and Wang, Mallory. 2019. “Dynamic Ecological Inference for Time-Varying Population Distributions Based on Sparse, Irregular, and Noisy Marginal Data.” Political Analysis. 27 (3): 388–396. https://doi.Org/10.1017/pan.2019.4.Google Scholar
Caughey, Devin, and Warshaw, Christopher. 2019. “Public Opinion in Subnational Politics.” Journal of Politics 81 (Symposium on Subnational Policymaking): 352–363.Google Scholar
Cochran, William G. 1977. Sampling Techniques. 3rd ed. New York: Wiley.Google Scholar
Converse, Jean M. 1987. Survey Research in the United States: Roots and Emergence. Berkeley: University of California Press.Google Scholar
Davison, A. C., and Hinkley, David V.. 1997. Bootstrap Methods and Their Application. New York: Cambridge University Press.Google Scholar
Deming, W. Edwards, and Stephan, F. Frederick. 1940. “On a Least Squares Adjustment of a Sampled Frequency Table When the Expected Marginal Totals Are Known.” Annals of Mathematical Statistics 11 (4): 427–444.Google Scholar
Dever, Jill A., and Valliant, Richard. 2010. “A Comparison of Variance Estimators for Poststratification to Estimated Control Totals.” Survey Methodology 36 (1): 45–56.Google Scholar
Dever, Jill A., and Valliant, Richard. 2016. “General Regression Estimation Adjusted for Undercoverage and Estimated Control Totals.” Journal of Survey Statistics and Methodology 4 (3): 289–318.Google Scholar
Deville, Jean-Claude. 1991. “A Theory of Quota Surveys.” Survey Methodology 17 (2): 163–181.Google Scholar
Deville, Jean-Claude. 2000. “Simultaneous Calibration of Several Surveys.” In Combining Data from Different Sources, Proceedings of Statistics Canada Symposium 99, 207–212. Ottawa: Statistics Canada.Google Scholar
Deville, Jean-Claude, and Sarndal, Carl-Erik. 1992. “Calibration Estimators in Survey Sampling.” Journal of the American Statistical Association 87 (418): 376–382.Google Scholar
Dixon, John, and Tucker, Clyde. 2010. “Survey Nonresponse.” In Handbook of Survey Research, 2nd ed., edited by Marsden, Peter V. and Wright, James D., 593–630. Bingley, UK: Emerald.Google Scholar
Economic Behavior Program, Survey Research Center, University of Michigan. 1948. Survey of Consumer Finances. Ann Arbor, MI: Institute for Social Research, Social Science Archive [producer], 1973. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], 2002. http://doi.org/10.3886/ICPSR03601.v1.Google Scholar
Elliott, Michael R., and Roderick, J. A. Little. 2000. “Model-Based Alternatives to Trimming Survey Weights.” Journal of Official Statistics 16 (3): 191–209.Google Scholar
Enns, Peter K., and Koch, Julianna. 2013. “Public Opinion in the U.S. States: 1956 to 2010.” State Politics & Policy Quarterly 13 (3): 349–372.Google Scholar
Field, Alexander J. 2006. “Table Dg34–45: Telephone industry - telephones, access lines, wire, employees, and plant: 1876–2000.” In Historical Statistics of the United States, Earliest Times to the Present: Millennial Edition, edited by Carter, Susan B., Gartner, Scott Sigmund, Haines, Michael R., Olmstead, Alan L., Sutch, Richard, and Wright, Gavin. New York: Cambridge University Press. http://hsus.cambridge.org/SeriesDg8–116.Google Scholar
Folger, John K., and Charles, B. Nam. 1964. “Educational Trends from Census Data.” Demography 1 (1): 247–257.Google Scholar
Freedman, David A. 2001. “Ecological Inference and the Ecological Fallacy.” In International Encyclopaedia of the Social and Behavioural Sciences, edited by Smelser, N. J., and Baltes, P. B., 6:4027–4030. New York: Elsevier.Google Scholar
Gelman, Andrew. 2007. “Struggles with Survey Weighting and Regression Modeling.” Statistical Science 22 (2): 153–164.Google Scholar
Gelman, Andrew, and John, B. Carlin. 2002. “Poststratification and Weighting Adjustments.” In Survey Nonresponse, edited by Graves, Robert M., Dillman, Don A., Eltinge, John L., and Little, Roderick J. A., 289–302. New York: Wiley.Google Scholar
Gelman, Andrew, and Thomas, C. Little. 1997. “Poststratification into Many Categories Using Hierarchical Logistic Regression.” Survey Methodology 23 (2): 127–135.Google Scholar
Groves, Robert M. 2006. “Nonresponse Rates and Nonresponse Bias in Household Surveys.” Public Opinion Quarterly 70 (5): 646–675.Google Scholar
Guandalini, Alessio, and Yves, Tillé. 2017. “Design-Based Estimators Calibrated on Estimated Totals from Multiple Surveys.” International Statistical Review 85 (2): 250–269.Google Scholar
Hainmueller, Jens. 2012. “Entropy Balancing for Causal Effects: A Multivariate Reweighting Method to Produce Balanced Samples in Observational Studies.” Political Analysis 20 (1): 25–46.Google Scholar
Hájek, Jaroslav. 1958. “On the Theory of Ratio Estimates.” Applied Mathematics 3 (5): 384–398.CrossRefGoogle Scholar
Hartman, Erin, Grieve, Richard, Ramsahai, Roland, and Sekhon, Jasjeet S.. 2015. “From Sample Average Treatment Effect to Population Average Treatment Effect on the Treated: Combining Experimental with Observational Studies to Estimate Population Treatment Effects.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 178 (3): 757–778.Google Scholar
Hillygus, Sunshine. 2016. “The Practice of Survey Research: Changes and Challenges.” In New Directions in Public Opinion, 2nd ed., edited by Berinsky, Adam J., 34–53. New York: Routledge.Google Scholar
Horvitz, D. G., and Thompson, D. J.. 1952. “A Generalization of Sampling without Replacement from a Finite Universe.” Journal of the American Statistical Association 47: 663–685.Google Scholar
Hur, Aram, and Achen, Christopher H.. 2013. “Coding Voter Turnout Responses in the Current Population Survey.” Public Opinion Quarterly 77 (4): 985–993.Google Scholar
Ireland, C. T., and Kullback, S.. 1968. “Contingency Tables with Given Marginals.” Biometrika 55 (1): 179–188.Google Scholar
Kalton, Graham, and Flores-Cervantes, Ismael. 2003. “Weighting Methods.” Journal of Official Statistics 19 (2): 81–97.Google Scholar
Kastellec, Jonathan P, Lax, Jeffrey R., Malecki, Michael, and Phillips, Justin H.. 2015. “Polarizing the Electoral Connection: Partisan Representation in Supreme Court Confirmation Politics.” Journal of Politics 77 (3): 787–804.Google Scholar
Kennedy, Courtney, Blumenthal, Mark, Clement, Scott et al. 2018. “An Evaluation of the 2016 Election Polls in the United States.” Public Opinion Quarterly 82 (1): 1–33.Google Scholar
Key, V O. Jr. 1949. Southern Politics in State and Nation. New York: Knopf.Google Scholar
King, Gary, Rosen, Ori, and Tanner, Martin A.. 2004. “Information in Ecological Inference: An Introduction.” In Ecological Inference: New Methodological Strategies, edited by King, Gary, Rosen, Ori, and Tanner, Martin A., 1–12. New York: Cambridge University Press.Google Scholar
Kish, Leslie. 1965. Survey Sampling. New York: Wiley.Google Scholar
Kott, Phillip S. 2006. “Using Calibration Weighting to Adjust for Nonresponse and Coverage Errors.” Survey Methodology 32 (2): 133–142.Google Scholar
Lax, Jeffrey R., and Justin, H. Phillips. 2009. “How Should We Estimate Public Opinion in The States?” American Journal of Political Science 53 (1): 107–121.CrossRefGoogle Scholar
Leeman, Lucas, and Wasserfallen, Fabio. 2017. “Extending the Use and Prediction Precision of Subnational Public Opinion Estimation.” American Journal of Political Science 61 (4): 1003–1022.Google Scholar
Leeuw, E. de, and Heer, W. de. 2002. “Trends in Household Survey Nonresponse: A Longitudinal and International Comparison.” In Survey Nonresponse, edited by Groves, R. M., Dillman, D. A., Eltinge, J. L., and Little, R. J. A., 41–54. New York: Wiley.Google Scholar
Little, R. J. A. 1993. “Post-Stratification: A Modeler’s Perspective.” Journal of the American Statistical Association 88 (423): 1001–1012.Google Scholar
Little, Roderick J. A., and Wu, Mei-Miau. 1991. “Models for Contingency Tables With Known Margins When Target and Sampled Populations Differ.” Journal of the American Statistical Association 86 (413): 87–95.Google Scholar
Little, Roderick J., and Vartivarian, Sonya. 2005. “Does Weighting for Nonresponse Increase the Variance of Survey Means?” Survey Methodology 31 (2): 161–168.Google Scholar
Luevano, Patricia. 1994. Response Rates in the National Election Studies, 1948–1992. Technical report 44. March. www.electionstudies.org/Library/papers/documents/nes010162.pdf.Google Scholar
Lumley, Thomas S. 2010. Complex Surveys: A Guide to Analysis Using R. Hoboken, NJ: Wiley.Google Scholar
Mackuen, Michael B., Erikson, Robert S., and Stimson, James A.. 1989. “Macropartisanship.” American Political Science Review 83 (4): 1125–1142.Google Scholar
Marken, Stephanie. 2018. “Still Listening: The State of Telephone Surveys.” Gallup, Methodology Blog, January 18. http://news.gallup.com/opinion/methodology/225143/listening–state–telephone–surveys.aspx.Google Scholar
McDonald, Michael. 2019. “CPS Vote Over-Report and Non-Response Bias Correction.” Accessed October 2. http://www.electproject.org/home/voter-turnout/cps-methodology.Google Scholar
Mickey, Robert W. 2015. Paths Out of Dixie: The Democratization of Authoritarian Enclaves in America’s Deep South. Princeton, NJ: Princeton University Press.Google Scholar
Miratrix, Luke, Sekhon, Jasjeet S., Theodoridis, Alexander G., and Campos, Luis F.. 2018. “Worth Weighting? Howto Think About and Use Weights in Survey Experiments.” Political Analysis 26: 275–291.Google Scholar
Mosteller, Frederick, Hyman, Herbert, McCarthy, Philip J., Marks, Eli S., and Truman, David B., eds. 1949. The Pre-Election Polls of 1948. New York: Social Science Research Council.Google Scholar
Moy, Corrine. 2015. “Fit-for-purpose sampling.” International Journal of Market Research 57 (3): 491–494.Google Scholar
Norpoth, Helmut, Sidman, Andrew H., and Suong, Clara H.. 2013. “Polls and Elections: The New Deal Realignment in Real Time.” Presidential Studies Quarterly 43 (1): 146–166.Google Scholar
Park, David K., Gelman, Andrew, and Bafumi, Joseph. 2004. “Bayesian Multilevel Estimation with Poststratification: State-Level Estimates from National Polls.” Political Analysis 12 (4): 375–385.Google Scholar
Pew, Research Center. 2016. As Election Nears, Voters Divided Over Democracy and “Respect”. Technical report. October. www.people–press.org/2016/10/27/as–election–nears–voters–divided–over–democracy–and–respect/.Google Scholar
Peytchev, Andy. 2012. “Multiple Imputation for Unit Nonresponse and Measurement Error.” Public Opinion Quarterly 76 (2): 214–237.Google Scholar
Quinn, Kevin M. 2004. “Ecological Inference in the Presence of Temporal Dependence.” In Ecological Inference: New Methodological Strategies, edited by King, Gary, Rosen, Ori, and Tanner, Martin A., 207–233. New York: Cambridge University Press.Google Scholar
Core Team, R. 2018. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. www.R–project.org/.Google Scholar
Rubin, Donald B. 1974. “Estimating Causal Effects of Treatments in Randomized and Non-randomized Studies.” Journal of Educational Psychology 66 (5): 688–701.Google Scholar
Ruggles, Steven, Alexander, J. Trent, Genadek, Katie, Goeken, Ronald, Schroeder, Matthew B., and Sobek, Matthew. 2010. Integrated Public Use Microdata Series: Version 5.0 [Machine-readable database]. Minneapolis: University of Minnesota.Google Scholar
Ruggles, Steven, Genadek, Katie, Goeken, Ronald, Grover, Josiah, and Sobek, Matthew. 2017. Integrated Public Use Microdata Series: Version 7.0 [dataset]. Minneapolis: University of Minnesota. https://doi.Org/10.18128/D010.V7..Google Scholar
Särndal, Carl-Erik, and Lundstrom, Sixten. 2005. Estimation in Surveys with Nonresponse. Hoboken, NJ: Wiley.Google Scholar
Särndal, Carl-Erik, and Lundstrom, Sixten. 2008. “Assessing Auxiliary Vectors for Control of Nonresponse Bias in the Calibration Estimator.” Journal of Official Statistics 24 (2): 167–191.Google Scholar
Särndal, Carl-Erik, and Traat, Imbi. 2011. “Domain Estimators Calibrated on Information from Another Survey.” Acta et Commentationes Universitatis Tartuensis de Mathematics 15 (2): 43–60.Google Scholar
Schickler, Eric, and Caughey, Devin. 2011. “Public Opinion, Organized Labor, and the Limits of New Deal Liberalism, 1936–1945.” Studies in American Political Development 25 (2): 1–28.Google Scholar
Splawa-Neyman, Jerzy. 1923. “On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.” Roczniki Nauk Roiniczych, Tom X: 1–51. Reprinted in Statistical Science, 5 (4): 465–472, 1990. Translated from Polish by D. M. Dabrowska and T. P. Speed.Google Scholar
Squire, Peverill. 1988. “Why the 1936 Literary Digest Poll Failed.” Public Opinion Quarterly 52: 125–133.Google Scholar
Stuart, Elizabeth A., Cole, Stephen R., Bradshaw, Catherine P., and Leaf, Philip J.. 2011. “The Use of Propensity Scores to Assess the Generalizability of Results from Randomized Trials.” Journal of the Royal Statistical Society. Series A (Statistics in Society) 174 (2): 369–386.Google Scholar
Tanner, Martin A. 1996. Tools for Statistical Inference Methods for the Exploration of Posterior Distributions and Likelihood Functions. 3rd ed. New York: Springer-Verlag.Google Scholar
Tibshirani, Robert. 1996. “Regression Shrinkage and Selection via the Lasso.” Journal of the Royal Statistical Society. Series B (Methodological) 58 (1): 267–288.Google Scholar
Treier, Shawn, and Jackman, Simon. 2008. “Democracy as a Latent Variable.” American Journal of Political Science 52 (1): 201–217.CrossRefGoogle Scholar
Valliant, Richard, Dever, Jill A., and Kreuter, Frauke. 2018. Practical Tools for Designing and Weighting Survey Samples. Springer. PDF e-book.Google Scholar
Verba, Sidney, and Lehman Schlozman, Kay. 1977. “Unemployment, Class Consciousness, and Radical Politics: What Didn’t Happen in the Thirties.” Journal of Politics 39 (2): 291–323.Google Scholar
Wagner, James. 2012. “A Comparison of Alternative Indicators for the Risk of Nonresponse Bias.” Public Opinion Quarterly 76 (3): 555–575.Google Scholar
Wakefield, Jon. 2004. “Ecological Inference for 2 × 2 Tables.” Journal of the Royal Statistical Society. Series A (General) 167 (3): 385–445.Google Scholar
Warshaw, Christopher, and Rodden, Jonathan. 2012. “How Should We Measure District-Level Public Opinion on Individual Issues?” The Journal of Politics 74 (1): 203–219.Google Scholar
Weatherford, M. Stephen, and Sergeyev, Boris. 2000. “Thinking about Economic Interests: Class and Recession in the New Deal.” Political Behavior 22 (4): 311–339.Google Scholar
Wiseman, Frederick, and McDonald, Philip. 1979. “Noncontact and Refusal Rates in Consumer Telephone Surveys.” Journal of Marketing Research 16 (4): 478–484.Google Scholar

Save element to Kindle

To save this element to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Target Estimation and Adjustment Weighting for Survey Nonresponse and Sampling Bias
Available formats
×

Save element to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Target Estimation and Adjustment Weighting for Survey Nonresponse and Sampling Bias
Available formats
×

Save element to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Target Estimation and Adjustment Weighting for Survey Nonresponse and Sampling Bias
Available formats
×