Hostname: page-component-78c5997874-8bhkd Total loading time: 0 Render date: 2024-11-13T01:56:57.588Z Has data issue: false hasContentIssue false
  • English
  • Français

Extracting Wisdom from Experts and Small Crowds: Strategies for Improving Informant-based Measures of Political Concepts

Published online by Cambridge University Press:  04 January 2017

Cherie D. Maestas ,
Matthew K. Buttice  and
Walter J. Stone
Cherie D. Maestas*
Affiliation:
Department of Political Science, Florida State University, 531 Bellamy, Tallahassee, FL 32306
Matthew K. Buttice
Affiliation:
California Research Bureau, California State Library, Sacramento, CA 94237-0001. e-mail: matthew.buttice@library.ca.gov
Walter J. Stone
Affiliation:
Department of Political Science, University of California, Davis, One Shields Ave., Davis, CA 95616. e-mail: wstone@ucdavis.edu
*
e-mail: cmaestas@fsu.edu (corresponding author)
Get access Rights & Permissions [Opens in a new window]

Abstract

Social scientists have increasingly turned to expert judgments to generate data for difficult-to-measure concepts, but getting access to and response from highly expert informants can be costly and challenging. We examine how informant selection and post-survey response aggregation influence the validity and reliability of measures built from informant observations. We draw upon three surveys with parallel survey questions of candidate characteristics to examine the trade-off between expanding the size of the local informant pool and the pool's level of expertise. We find that a “wisdom-of-crowds” effect trumps the benefits associated with the expertise of individual informants when the size of the rater pool is modestly increased. We demonstrate that the benefits of expertise are best realized by prescreening potential informants for expertise rather than post-survey weighting by expertise.

Type
Symposium on Advances in Survey Methodology
Information
Political Analysis , Volume 22 , Issue 3 , Summer 2014 , pp. 354 - 373
Copyright
Copyright © The Author 2014. Published by Oxford University Press on behalf of the Society for Political Methodology 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Authors' note: We would like to thank Lonna Rae Atkeson, Alex Adams, Ben Highton, Brad Jones, Chris Reenock, and the anonymous reviewers for helpful comments on previous drafts. Matthew Buttice began work on this project while at UC Davis and finished while at the California Research Bureau. The research results and conclusions expressed in this article do not necessarily reflect the views of the California Research Bureau or California State Library. Supplementary materials for this article are available on the Political Analysis Web site.

References

Albright, Jeremy J., and Mair, Peter. 2011. Does the number of parties to place affect the placement of parties? Results from an expert survey experiment. Electoral Studies 30 (4): 858-64.CrossRefGoogle Scholar
Andersson, Patric, Edman, Jan, and Ekman, Mattais. 2005. Predicting the World Cup 2002: Performance and confidence of experts and non-experts. International Journal of Forecasting 21 (3): 565-76.Google Scholar
Ansolabehere, Stephen. 2010. CCES Common Content, 2010. http://hdl.handle.net/1902.1/17705 V3 [Version] (accessed January 18, 2014).Google Scholar
Benoit, Kenneth, and Laver, Michael. 2006. Party policy in modern democracies. London: Routledge.Google Scholar
Boyer, Kenneth K., and Verma, Rohit. 2000. Multiple raters in survey-based operations management research: A review and tutorial. Production and Operations Management 9 (2): 128-40.CrossRefGoogle Scholar
Braouezec, Yann. 2010. Committee, expert advice, and the weighted majority algorithm: An application to the pricing decision of a monopolist. Computational Economics 35: 245-67.Google Scholar
Budge, Ian. 2000. Expert judgments of party policy positions: Uses and limitations in political research. European Journal of Political Research 37: 103-13.Google Scholar
Buttice, Matthew K., and Stone, Walter J. 2012. Candidates matter: Policy and quality differences in congressional elections. Journal of Politics 74 (3): 870-87.Google Scholar
Carroll, Royce, Lewis, Jeffrey B., Lo, James, Poole, Keith T., and Rosenthal, Howard. 2009. Measuring bias and uncertainty in DW-NOMINATE ideal point estimates via the parametric bootstrap. Political Analysis 17 (3): 261-75.Google Scholar
Castles, Francis G., and Mair, Peter. 1984. Left-right political scales, some “experts” judgments. European Journal of Political Research 12 (1): 73–88.Google Scholar
Clinton, Joshua D., and Lewis, David E. 2008. Expert opinion, agency characteristics, and agency preferences. Political Analysis 16 (1): 3–20.Google Scholar
Cooke, Roger M., and Goossens, Louis H. J. 2004. Expert judgement elicitation for risk assessments of critical infrastructures. Journal of Risk Research 7 (6): 643-56.Google Scholar
Dancy, Logan, and Sheagley, Geoffrey. 2013. Heuristics behaving badly: Party cues and voter knowledge. American Journal of Political Science 57 (2): 312-25.Google Scholar
Feldman, Stanley, and Conover, Pamela. 1983. Candidates, issues and voters: The role of inference in political perception. Journal of Politics 45 (4): 810-39.Google Scholar
Gaissmaier, Wolfgang, and Marewski, Julian N. 2011. Forecasting elections with mere recognition from small lousy samples: A comparison of collective recognition, wisdom of crowds, and representative polls. Judgment and Decision Making 6 (1): 73–88.Google Scholar
Hooghe, Liesbet, Bakker, Ryan, Brigevich, Anna, De Vries, Catherine, Edwards, Erica, Marks, Gary, Rovny, Jan, Steenbergen, Marco, and Vachudova, Milada. 2010. Reliability and validity of the 2002 and 2006 Chapel Hill Expert Surveys on party positioning. European Journal of Political Research 49: 687–703.Google Scholar
Huber, John, and Inglehart, Richard. 1995. Expert interpretations of party space and party locations in 42 societies. Party Politics 1 (1): 73–111.CrossRefGoogle Scholar
Huckfeldt, Robert, Carmines, Edward G., Mondak, Jeffery J., and Zeemering, Eric. 2007. Information, activation, and electoral competition in the 2002 congressional elections. Journal of Politics 69 (3): 798–812.Google Scholar
Javaras, Kristin N., Hill Godlsmith, H., and Laird, Nan M. 2011. Estimating the effect of a predictor measured by two informants on a continuous outcome: A comparison of methods. Epidemiology 22 (3): 390-99.Google Scholar
Jones, Bradford S., and Norrander, Barbara. 1996. The reliability of aggregated public opinion measures. American Journal of Political Science 40 (2): 295–309.CrossRefGoogle Scholar
Kitschelt, Herbert, and Kselman, Daniel M. 2012. Economic development, democratic experience, and political parties linkage strategies. Comparative Political Studies September:1–32.Google Scholar
Lorenz, Jan, Rauhut, Heiko, Schweitzer, Frank, and Helbing, Dirk. 2011. How social influence can undermine the wisdom of crowds effect. Proceedings of the National Acadamy of Science 108 (22): 9020-25.Google Scholar
Maestas, Cherie D., Buttice, Matthew K., and Stone, Walter J. 2013. Replication data for: Extracting wisdom from experts and small crowds: Strategies for improving informant-based measures of political concepts. http://dx.doi.org/10.7910/DVN/23170 IQSS Dataverse Network [Distributor] V1 [Version] (accessed January 18, 2014).Google Scholar
Martin, Tara G., Bergman, Mark A., Fidler, Fiona, Kuhnert, Petra M., Low-Choy, Samantha, McBride, Marissa, and Mengersen, Kerrie. 2012. Eliciting expert knowledge in conservation science. Conservation Biology 26 (1): 29–38.Google Scholar
McCann, Stewart J. H. 1998. The extended American social, economic, and political threat index (1788–1992). Journal of Psychology 132 (4): 435-49.Google Scholar
Page Scott, E. 2007. The difference: How the power of diversity creates better groups, firms, schools, and societies. Princeton, NJ: Princeton University Press.Google Scholar
Phillips, Lynn W. 1981. Assessing measurement error in key informant reports: A methodological note on organizational analysis in marketing. Journal of Marketing Research 18 (4): 395–415.CrossRefGoogle Scholar
Poole, Keith T., and Rosenthal, Howard. 1997. Congress: A political-economic history of roll call voting. New York: Oxford University Press.Google Scholar
Powell, Lynda. 1989. Analyzing misinformation: Perceptions of congressional candidates’ ideologies. American Journal of Political Science 33: 272-93.CrossRefGoogle Scholar
Saiegh, Sebastian M. 2009. Recovering a basic space from elite surveys: Evidence from Latin America. Legislative Studies Quarterly 34 (1): 117-45.Google Scholar
Sjoberg, Lennart. 2009. Are all crowds equally wise? A comparison of political election forecasts by experts and the public. Journal of Forecasting 28 (1): 1–18.Google Scholar
Steenbergen, Marco, and Marks, Gary. 2007. Evaluating expert judgments. European Journal of Political Research 46: 347-66.Google Scholar
Stone, Walter J., and Simas, Elizabeth N. 2010. Candidate valence and ideological positions in U.S. House elections. American Journal of Political Science 54 (2): 371-88.Google Scholar
Stone, Walter J., Fulton, Sarah A., Maestas, Cherie D., and Sandy Maisel, L. 2010. Incumbency reconsidered: Prospects, strategic retirement, and incumbent quality in U.S. House elections. Journal of Politics 72 (1): 178-90.Google Scholar
Stone, Walter J., Sandy Maisel, L., and Maestas, Cherie D. 2004. Quality counts: Extending the strategic politician model of incumbent deterrence. American Journal of Political Science 48 (3): 479-95.Google Scholar
Surowiecki, Jame. 2004. The wisdom of crowds. New York: Random House.Google Scholar
Van Bruggen, Gerrit H., Lilien, Gary L., and Kacker, Manish. 2002. Informants in organizational marketing research: Why use multiple informants and how to aggregate responses. Journal of Marketing Research 39 (4): 469-78.Google Scholar
Wagner, Stephan M., Rau, Christian, and Lindemann, Eckhard. 2010. Multiple informant methodology: A critical review and recommendations. Sociological Methods and Research 38 (4): 582–618.Google Scholar
Whitefield, Stephen, Anna Vachudova, Matilda, Steenbergen, Marco R., Rohrschneider, Robert, Marks, Gary, Loveless, Matthew P., and Hooghe, Liesbet. 2007. Do expert surveys produce consistent estimates of party stances on European integration? Comparing expert surveys in the difficult case of Central and Eastern Europe. Electoral Studies 26 (1): 50–61.Google Scholar
Supplementary material: File

Maestas et al. supplementary material

Appendix

Download Maestas et al. supplementary material(File)
File 43 KB