Hostname: page-component-78c5997874-4rdpn Total loading time: 0 Render date: 2024-11-11T06:23:27.178Z Has data issue: false hasContentIssue false
  • English
  • Français

Learning to See: Convolutional Neural Networks for the Analysis of Social Science Data

Published online by Cambridge University Press:  16 April 2021

Michelle Torres Open the ORCID record for Michelle Torres [Opens in a new window]  and
Francisco Cantú
Michelle Torres*
Affiliation:
Rice University, Houston, TX, USA. E-mail: smtorres@rice.edu
Francisco Cantú
Affiliation:
University of Houston, Houston, TX, USA. E-mail: fcantu10@uh.edu
*
Corresponding author Michelle Torres
Get access Rights & Permissions [Opens in a new window]

Abstract

We provide an introduction of the functioning, implementation, and challenges of convolutional neural networks (CNNs) to classify visual information in social sciences. This tool can help scholars to make more efficient the tedious task of classifying images and extracting information from them. We illustrate the implementation and impact of this methodology by coding handwritten information from vote tallies. Our paper not only demonstrates the contributions of CNNs to both scholars and policy practitioners, but also presents the practical challenges and limitations of the method, providing advice on how to deal with these issues.

Keywords

Image analysisdeep learningneural networkcomputer vision
Type
Article
Information
Political Analysis , Volume 30 , Issue 1 , January 2022 , pp. 113 - 131
Copyright
© The Author(s) 2021. Published by Cambridge University Press on behalf of the Society for Political Methodology

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Edited by Lonna Atkeson

References

Anastasopoulos, L. J., Badani, D., Lee, C., Ginosar, S., and Williams, J.. 2016. “Photographic Home Styles in the House and Senate: A Computer Vision Approach.” Working Paper.Google Scholar
Barberá, P. 2015. “Birds of the Same Feather Tweet Together. Bayesian Ideal Point Estimation Using Twitter Data.” Political Analysis 23(1):76–91.CrossRefGoogle Scholar
Boureau, Y.-L., Ponce, J., and LeCun, Y.. 2010. “A Theoretical Analysis of Feature Pooling in Visual Recognition.” In Proceedings of the 27th International Conference on Machine Learning (ICML-10), 111–118. Haifa, Israel: Omnipress.Google Scholar
Boxell, L. 2020. “Slanted Images: Measuring Nonverbal Media Bias.” Working Paper.CrossRefGoogle Scholar
Buda, M., Maki, A., and Mazurowski, M. A.. 2018. “A Systematic Study of the Class Imbalance Problem in Convolutional Neural Networks.” Neural Networks 106:249–259.CrossRefGoogle ScholarPubMed
Buduma, N., and Locascio, N.. 2017. Fundamentals of Deep Learning: Designing Next-Generation Machine Intelligence Algorithms. Sebastopol, CA: O’Reilly Media.Google Scholar
Cantú, F. 2019. “The Fingerprints of Fraud: Evidence from Mexico’s 1988 Presidential Election.” American Political Science Review 113(3):710–726.CrossRefGoogle Scholar
Cantú, F., and Torres, M.. 2020a. “Replication Data for: Learning to See: Convolutional Neural Networks for the Analysis of Social Science Data.” Code Ocean. https://doi.org/10.24433/CO.3401138.v1.CrossRefGoogle Scholar
Cantú, F., and Torres, M.. 2020b. “Replication Data for: Learning to See: Convolutional Neural Networks for the Analysis of Social Science Data.” Harvard Dataverse p. V1. https://doi.org/10.7910/DVN/UNYOLF.CrossRefGoogle Scholar
Casas, A., and Webb Williams, N.. 2019. “Images that Matter: Online Protests and the Mobilizing Role of Pictures.” Political Research Quarterly 72(2):360–375.CrossRefGoogle Scholar
Challú, C., Seira, E., and Simpser, A.. 2020 “The Quality of Vote Tallies.” American Political Science Review 114(4):1071–1085.CrossRefGoogle Scholar
Chan, P. K., and Stolf, S. J.. 1998. “Toward Scalable Learning with Non-Uniform Class and Cost Distributions: A Case Study in Credit Card Fraud Detection.” In Proceeding of the Fourth International Conference on Knowledge Discovery and Data Mining, 164–168. New York: AAAI.Google Scholar
Chatfield, K., Simonyan, K., Vedaldi, A., and Zisserman, A.. 2014. “Return of the Devil in the Details: Delving Deep into Convolutional Nets.” In Proceedings of the British Machine Vision Conference. Nottingham, UK: BMVA Press.Google Scholar
Ciresan, D. C., Meier, U., Masci, J., Gambardella, L. M., and Schmidhuber, J.. 2011. “Flexible, High Performance Convolutional Neural Networks for Image Classification.” In Twenty-Second International Joint Conference on Artificial Intelligence, 1237–1242. Barcelona, Spain: AAAI.Google Scholar
Coüasnon, B., Camillerapp, J., and Leplumey, I.. 2007. “Access by Content to Handwritten Archive Documents: Generic Document Recognition Method and Platform for Annotations.” International Journal of Document Analysis and Recognition 9(2–4):223–242.CrossRefGoogle Scholar
Dietrich, B. J. 2020. “Using Motion Detection to Measure Social Polarization in the U.S. House of Representatives.” Political Analysis, doi: 10.1017/pan.2020.25.CrossRefGoogle Scholar
Dietrich, B. J., Enos, R. D., and Sen, M.. 2019. “Emotional Arousal Predicts Voting on the US Supreme Court.” Political Analysis 27(2):237–243.CrossRefGoogle Scholar
Dietrich, B. J, Hayes, M., and O’Brien, D.. 2019. “Pitch Perfect: Vocal Pitch and the Emotional Intensity of Congressional Speech.” American Political Science Review 113(4):941–962.CrossRefGoogle Scholar
Gong, Y., Wang, L., Guo, R., and Lazebnik, S.. 2014. “Multi-Scale Orderless Pooling of Deep Convolutional Activation Features.” In European Conference on Computer Vision, 392–407. Zurich, Switzerland: Springer.Google Scholar
Goodfellow, I., Bengio, Y., and Courville, A.. 2016. Deep Learning. Cambridge, MA: MIT Press.Google Scholar
Grimmer, J., and Stewart, B.. 2013. “Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts.” Political Analysis 21(3):267–297.CrossRefGoogle Scholar
Han, S. et al. 2018. “Optimizing Filter Size in Convolutional Neural Networks for Facial Action Unit Recognition”. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5070–5078. Los Alamitos, CA: IEEE Computer Society.CrossRefGoogle Scholar
Homola, J. 2018. “The Political Consequences of Group-Based Identities.” Working Paper.Google Scholar
Huang, S.-J., Jin, R., and Zhou, Z.-H.. 2014. “Active Learning by Querying Informative and Representative Examples.” IEEE Transactions on Pattern Analysis and Machine Intelligence 36(10):1936–1949.CrossRefGoogle ScholarPubMed
Huff, C. D. 2018. “Why Rebels Reject Peace.” Working Paper.Google Scholar
Ioffe, S., and Szagedy, C.. 2015. “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.” In International Conference on Machine Learning . Lille, France: PMLR.Google Scholar
Japkowicz, N., and Stepehn, S.. 2002. “The Class Imbalance Problem: A Systematic Study.” Intelligent Data Analysis 6(5):429–449.CrossRefGoogle Scholar
Kiranyaz, S., Avci, O., Abdeljaber, O., Ince, T., Gabbouj, M., and Inman, D. J.. 2021. “1D Convolutional Neural Networks and Applications—A Survey.” Mechanical Systems and Signal Processing 151:107398.CrossRefGoogle Scholar
Kubat, M., Holte, R. C., and Matwin, S.. 1998. “Machine Learning for the Detection of Oil Spills in Satellite Radar Images.” Machine Learning 30(2–3):195–215.CrossRefGoogle Scholar
LeCun, Y. et al. 1989. “Backpropagation Applied to Handwritten Zip Code Recognition.” Neural Computation 1(4):541–551.CrossRefGoogle Scholar
Lee, C.-Y., Gallagher, P. W., and Tu, Z..2016. “Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree.” In Artificial Intelligence and Statistics, 464–472. Cadiz, Spain: PMLR.Google Scholar
Lipton, Z. C. 2016. “The Mythos of Model Interpretability.” In 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016). New York.Google Scholar
Lladós, J., Pratim-Roy, P., Rodríguez, J. A, and Sánchez, G.. 2007. “Word Spotting in Archive Documents Using Shape Contexts.” In Iberian Conference on Pattern Recognition and Image Analysis, 290–297. Faro, Portugal: Springer.CrossRefGoogle Scholar
Lucas, C. 2018a. “Neural Networks for the Social Sciences.” Working Paper.Google Scholar
Lucas, C. 2018b. “A Supervised Method for Automated Classification of Political Video.” Working Paper.Google Scholar
McCarty, N., Poole, K. T., and Rosenthal, H.. 2006. Polarized America. Cambridge, MA: The MIT Press.Google Scholar
Miller, B., Linder, F., and Mebane, W. R.. 2020. “Active Learning Approaches for Labeling Text: Review and Assessment of the Performance of Active Learning Approaches.” Political Analysis 28(4):532–551.CrossRefGoogle Scholar
Nair, V., and Hinton, G. E.. 2010. “Rectified Linear Units Improve Restricted Boltzmann Machines.” In ICML’10 Proceedings of the 27th International Conference on International Conference on Machine Learning, edited by Fürnkranz, J. and Joachims, T., 807–814. Haifa, Israel: Omnipress.Google Scholar
Neumann, M. 2019. “Fair and Balanced? News Media Bias in the Photographic Coverage of the 2016 U.S. Presidential Election.” Working Paper.Google Scholar
Nguyen, A., Yosinski, J., and Clune, J.. 2015. “Deep Neural Networks Are Easily Fooled: High Confidence Predictions for Unrecognizable Images.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 427–436. Boston, MA: IEEE.Google Scholar
Pan, S. J., and Yang, Q.. 2010. “A Survey on Transfer Learning.” IEEE Transactions on Knowledge and Data Engineering 22(10):1345–1359.CrossRefGoogle Scholar
Qin, Z., Yu, F., Liu, C., and Chen, X.. 2018. “How Convolutional Neural Networks See the World—A Survey of Convolutional Neural Network Visualization Methods.” Mathematical Foundations of Computing 1(2):149–180.CrossRefGoogle Scholar
Ranzato, M., Boureau, Y.-L., and Cun, Y. L.. 2008. “Sparse Feature Learning for Deep Belief Networks.” Advances in Neural Information Processing Systems 20:1185–1192.Google Scholar
Rosebrock, A. 2017. Deep Learning for Computer Vision with Python: Starter Bundle. PyImageSearch.Google Scholar
Ruder, S. 2017. “An Overview of Gradient Descent Optimization Algorithms.” arXiv:1609.04747.Google Scholar
Rumelhart, D. E., Hinton, G. E., and Williams, R. J.. 1986. “Learning Representations by Back-Propagating Errors.” Nature 23:533–536.CrossRefGoogle Scholar
Sabour, S., Frosst, N., and Hinton, G. E. 2017. “Dynamic Routing Between Capsules.” In Advances in Neural Information Processing Systems, 3856–3866. Long Beach, CA: NIPS.Google Scholar
Schrodt, P. A. 2004. “Patterns, Rules and Learning: Computational Models of International Behavior.” Working Paper.Google Scholar
Settles, B. 2009. “Active Learning Literature Survey.” Working Paper.Google Scholar
Shang, W., Sohn, K., Almeida, D., and Lee, H.. 2016. “Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units.” In Proceedings of the 33rd International Conference on International Conference on Machine Learning , vol. 48, 2217–2225. New York: PMLR.Google Scholar
Simonyan, K., and Zisserman, A.. 2015. “Very Deep Convolutional Networks for Large-Scale Image Recognition.” arXiv:1409.1556.Google Scholar
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R.. 2014. “Dropout: A Simple Way to Prevent Neural Networks from Overfitting.” Journal of Machine Learning Research 15:1929–1958.Google Scholar
Stein, R. M. et al. 2020. “Waiting to Vote in the 2016 Presidential Election: Evidence from a Multi-County Study.” Political Research Quarterly 73(2):439–453.CrossRefGoogle Scholar
Steinert-Threlkeld, Z., Joo, J., and Chan, A.. 2019. “How Violence Affects Protests.” Working Paper.CrossRefGoogle Scholar
Taylor, U. 2008. “Women in the Documents: Thoughts on Uncovering the Personal, Political, and Professional.” Journal of Women’s History 20(1):187–196.CrossRefGoogle Scholar
Webb Williams, N. 2019. “Automated Image Taggers from Amazon, Google, and Microsoft: Are They Useful for Social Science Research.” Working Paper.Google Scholar
Webb Williams, N., Casas, A., and Wilkerson, J. D.. 2020. Images as Data for Social Science Research: An Introduction to Convolutional Neural Nets for Image Classification . Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Wilcox-Archuleta, B. 2019. “Measuring Neighborhood Level Ethnic Visibility: Evidence from Street View Images.” Working Paper.Google Scholar
Won, D., Steinert-Threlkeld, Z. C., and Joo, J.. 2017. “Protest Activity Detection and Perceived Violence Estimation from Social Media Images.” In Proceedings of the 25th ACM International Conference on Multimedia, 786–794. Mountain View, CA: ACM.CrossRefGoogle Scholar
Zeiler, M. D., and Fergus, R.. 2014. “Visualizing and Understanding Convolutional Networks.” In European Conference on Computer Vision, 818–833. Zurich, Switzerland: Springer.Google Scholar
Zhang, H., and Pan, J.. 2019. “CASM: A Deep-Learning Approach for Identifying Collective Action Events with Text and Image Data from Social Media.” Sociological Methodology 49(1):1–57.CrossRefGoogle Scholar
Zhang, X., Wang, Y., and Shi, W.. 2018. “pCAMP: Performance Comparison of Machine Learning Packages on the Edges.” In USENIX Workshop on Hot Topics in Edge Computing (HotEdge 18), 1–6. Boston, MA: USENIX Association.Google Scholar
Supplementary material: PDF

Torres and Cantú supplementary material

Torres and Cantú supplementary material

Download Torres and Cantú supplementary material(PDF)
PDF 1.4 MB
Supplementary material: Link

Torres and Cantú Dataset

Dataset

https://doi.org/10.7910/DVN/UNYOLF
Link