Hostname: page-component-cd9895bd7-gbm5v Total loading time: 0 Render date: 2024-12-29T12:06:27.407Z Has data issue: false hasContentIssue false
  • English
  • Français

Guessing and Forgetting: A Latent Class Model for Measuring Learning

Published online by Cambridge University Press:  04 January 2017

M. Ken Cor  and
Gaurav Sood
M. Ken Cor*
Affiliation:
Faculty of Pharmacy and Pharmaceutical Sciences, 3-209 Edmonton Clinic Health Academy, University of Alberta, Edmonton, CA T5M1B3
Gaurav Sood
Affiliation:
e-mail: gsood07@gmail.com
*
e-mail: mcor@ualberta.ca
Get access Rights & Permissions [Opens in a new window]

Abstract

Guessing on closed-ended knowledge items is common. Under likely-to-hold assumptions, in the presence of guessing, the most common estimator of learning, difference between pre- and postprocess scores, is negatively biased. To account for guessing-related error, we develop a latent class model of how people respond to knowledge questions and identify the model with the mild assumption that people do not lose knowledge over short periods of time. A Monte Carlo simulation over a broad range of informative processes and knowledge items shows that the simple difference score is negatively biased and the method we develop here is unbiased. To demonstrate its use, we apply our model to data from Deliberative Polls. We find that estimates of learning, once adjusted for guessing, are about 13% higher. Adjusting for guessing also eliminates the gender gap in learning, and halves the pre-deliberation gender gap on political knowledge.

Type
Articles
Information
Political Analysis , Volume 24 , Issue 2 , Spring 2016 , pp. 226 - 242
Copyright
Copyright © The Author 2016. Published by Oxford University Press on behalf of the Society for Political Methodology 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Author's note: We thank Ed Haertel for his advice and encouragement, Robert Luskin for improving some of the language, and Pablo Barberá, Ying Cui, Kabir Khanna, Marc Meredith, and Pete Mohanty for useful comments. Replication files for this study are available on the Harvard Dataverse at http://dx.doi.org/10.7910/DVN/HZHVCU. See Cor and Sood (2016). Supplementary materials for this article are available on the Political Analysis Web site.

References

Alvarez, R. M., and Franklin, C. H. 1994. Uncertainty and political perceptions. Journal of Politics 56(3): 671–88.CrossRefGoogle Scholar
Alvarez, R. M., and Glasgow, G. 1996. Do voters learn from presidential election campaigns? Technical report.Google Scholar
Bartels, L. M. 1986. Issue voting under uncertainty: An empirical test. American Journal of Political Science 30(4): 709–28.CrossRefGoogle Scholar
Bransford, J. D., and Johnson, M. K. 1972. Contextual prerequisites for understanding: Some investigations of comprehension and recall. Journal of Verbal Learning and Verbal Behavior 11(6): 717–26.Google Scholar
Cooke, N. J., Atlas, R. S., Lane, D. M., and Berger, R. C. 1993. Role of high-level knowledge in memory for chess positions. American Journal of Psychology 106(3): 321–51.Google Scholar
Cor, K., and Sood, G. 2016. Replication data for: Guessing and forgetting: A latent class model for measuring learning. http://dx.doi.org/10.7910/DVN/HZHVCU, Harvard Dataverse, V1 (accessed February 3, 2016).Google Scholar
Cor, M. K. 2012. An experimental test of the effect of norm-referenced and criterion referenced feedback on just world beliefs, motivation, and performance: Does social disadvantage matter? PhD thesis, Stanford University, Stanford, CA.Google Scholar
Delli Carpini, M. X., and Keeter, S. 1996. What Americans know about politics and why it matters. New Haven: Yale University Press.Google Scholar
Dooling, D. J., and Lachman, R. 1971. Effects of comprehension on retention of prose. Journal of Experimental Psychology 88(2): 216.CrossRefGoogle Scholar
Eckhardt, B. B., Wood, M. R., and Jacobvitz, R. S. 1991. Verbal ability and prior knowledge contributions to adults’ comprehension of television. Communication Research 18(5): 636–49.CrossRefGoogle Scholar
Fishkin, J. 2009. When the people speak: Deliberative democracy and public consultation. Oxford: Oxford University Press.Google Scholar
Fishkin, J 2016. Center for deliberative polling, Stanford University. http://cdd.stanford.edu (accessed February 4, 2016).Google Scholar
Franklin, C. H. 1991. Eschewing obfuscation? Campaigns and the perception of US senate incumbents. American Political Science Review 85 (04): 1193–214.CrossRefGoogle Scholar
Frazer, E., and Macdonald, K. 2003. Sex differences in political knowledge in Britain. Political Studies 51(1): 67–83.CrossRefGoogle Scholar
Ghalanos, A., and Theussl, S. 2010. Rsolnp: General non-linear optimization. R package version 1–0. https://cran.r-project.org/web/packages/Rsolnp/Rsolnp.pdf.Google Scholar
Gibson, J. L., and Caldeira, G. A. 2009. Knowing the Supreme Court? A reconsideration of public ignorance of the high court. Journal of Politics 71(2): 429–41.CrossRefGoogle Scholar
Hambrick, D. Z. 2003. Why are some people more knowledgeable than others? A longitudinal study of knowledge acquisition. Memory & Cognition 31(6): 902–17.CrossRefGoogle ScholarPubMed
Hansen, E. A., Schmidt, F. L., and Hansen, J. C. 1975. A model for the correction for guessing on multiple-choice tests. ACM SIGSOC Bulletin 7(1–4): 24–8.CrossRefGoogle Scholar
Iyengar, S. 1990. Shortcuts to political knowledge: The role of selective attention and accessibility. Information and Democratic Processes, 160–85.Google Scholar
Lenz, G. S. 2009. Learning and opinion change, not priming: Reconsidering the priming hypothesis. American Journal of Political Science 53(4): 821–37.CrossRefGoogle Scholar
Lord, F. M. 1975. Formula scoring and number-right scoring. Journal of Educational Measurement 12(1): 7–11.CrossRefGoogle Scholar
Lord, F. M., Novick, M. R., and Birnbaum, A. 1968. Statistical theories of mental test scores. Addison-Wesley.Google Scholar
Luskin, R. C., and Bullock, J. G. 2004. Measuring political knowledge. Paper presented at the Annual Meeting of the Midwest Political Science Association.Google Scholar
Luskin, R. C., and Bullock, J. G. 2011. “Don't know” means “don't know”: DK responses and the public's level of political knowledge. Journal of Politics 73(2): 547–57.CrossRefGoogle Scholar
Luskin, R. C., Fishkin, J. S., and Jowell, R. 2002. Considered opinions: Deliberative polling in Britain. British Journal of Political Science 32(3): 455–87.Google Scholar
Luskin, R. C., and Sood, G. 2012. Hidden knowledge (and veiled ignorance): Is the public distinctly less (or even more) ignorant than we have thought? In Annual Meeting of the Midwest Political Science Association.Google Scholar
Luskin, R. C., Sood, G., and Helfer, A. 2014. Measuring learning in informative processes. Paper presented at the European Consortium of Political Science Research Workshop, Salamanca, Spain.Google Scholar
Mondak, J. J. 1999. Reconsidering the measurement of political knowledge. Political Analysis 8(1): 57–82.CrossRefGoogle Scholar
Mondak, J. J 2001. Developing valid knowledge scales. American Journal of Political Science, 224–38.CrossRefGoogle Scholar
Mondak, J. J., and Anderson, M. R. 2004. The knowledge gap: A reexamination of gender-based differences in political knowledge. Journal of Politics 66(2): 492–512.CrossRefGoogle Scholar
Nunnally, J. C. 1967. Psychometric theory. Tata McGraw-Hill Education.Google Scholar
O’Flynn, I., and Sood, G. 2014. What would Dahl say? An appraisal of the democratic credentials of deliberative polls and other mini-publics. In: Deliberative mini-publics, eds. Gronlund, K., Backtiger, A., and Setala, M., 41–58. Colchester: ECPR Press.Google Scholar
Pasek, J., Sood, G., and Krosnick, J. 2015. Misinformed about the affordable care act? Leveraging certainty to assess the prevalence of misinformation. Journal of Communication 65(4): 660–73.Google Scholar
Prior, M., and Lupia, A. 2008. Money, time, and political knowledge: Distinguishing quick recall and political learning skills. American Journal of Political Science 52(1): 169–83.CrossRefGoogle Scholar
Prior, M., Sood, G., and Khanna, K. 2015. You cannot be serious: The impact of accuracy incentives on partisan bias in reports of economic perceptions. Quarterly Journal of Political Science 10(4): 489–518.Google Scholar
Recht, D. R., and Leslie, L. 1988. Effect of prior knowledge on good and poor readers’ memory of text. Journal of Educational Psychology 80(1): 16.CrossRefGoogle Scholar
Sanchez, M. E., and Morchio, G. 1992. Probing “dont know” answers: Effects on survey estimates and variable relationships. Public Opinion Quarterly 56(4): 454–74.CrossRefGoogle Scholar
Sniderman, P. M., Brody, R. A., and Tetlock, P. E. 1993. Reasoning and choice: Explorations in political psychology. Cambridge University Press.Google Scholar
Stolle, D., and Gidengil, E. 2010. What do women really know? A gendered analysis of varieties of political knowledge. Perspectives on Politics 8(1): 93–109.CrossRefGoogle Scholar
Sturgis, P., Allum, N., and Smith, P. 2008. An experiment on the measurement of political knowledge in surveys. Public Opinion Quarterly 72(1): 90–102.CrossRefGoogle Scholar
Tallmadge, G. K. 1982. The correction for guessing a case in which its use made treatment effects appear larger than they really were. Evaluation Review 6(6), 837–41.CrossRefGoogle Scholar
Varadhan, R., and Grothendieck, G. 2011. Alabama: Constrained nonlinear optimization. R package version 1.Google Scholar
Verba, S., Burns, N., and Schlozman, K. L. 1997. Knowing and caring about politics: Gender and political engagement. Journal of Politics 59(4): 1051–72.Google Scholar
Supplementary material: PDF

Ken Cor and Sood supplementary material

Appendix

Download Ken Cor and Sood supplementary material(PDF)
PDF 330.2 KB