Matching for causal inference is a well-studied problem, but standard methods fail when the units to match are text documents: the high-dimensional and rich nature of the data renders exact matching infeasible, causes propensity scores to produce incomparable matches, and makes assessing match quality difficult. In this paper, we characterize a framework for matching text documents that decomposes existing methods into (1) the choice of text representation and (2) the choice of distance metric. We investigate how different choices within this framework affect both the quantity and quality of matches identified through a systematic multifactor evaluation experiment using human subjects. Altogether, we evaluate over 100 unique text-matching methods along with 5 comparison methods taken from the literature. Our experimental results identify methods that generate matches with higher subjective match quality than current state-of-the-art techniques. We enhance the precision of these results by developing a predictive model to estimate the match quality of pairs of text documents as a function of our various distance scores. This model, which we find successfully mimics human judgment, also allows for approximate and unsupervised evaluation of new procedures in our context. We then employ the identified best method to illustrate the utility of text matching in two applications. First, we engage with a substantive debate in the study of media bias by using text matching to control for topic selection when comparing news articles from thirteen news sources. We then show how conditioning on text data leads to more precise causal inferences in an observational study examining the effects of a medical intervention.

This article offers a description and discussion of “shadowing” as a data collection and analytic tool, highlighting potential research opportunities related to the direct observation of individuals—principally political elites—in their normal daily routine for an extended period of time, often between one day and one week. In contrast with large-scale data collection methods, including surveys, shadowing enables researchers to develop detailed observations of political behavior that are not limited by the availability of administrative data or the constraints of a questionnaire or an interview guide. Unlike more in-depth qualitative methods, such as ethnography, shadowing is scalable in a manner that allows for larger sample sizes and the potential for medium-N inference. I provide a detailed account of how to design and conduct a shadowing study, including sampling strategies, techniques for coding shadowing data, and processes for drawing inferences about the behavior of shadowed subjects, drawing on examples from a completed shadowing-based study. I also discuss ways to mitigate selection and observer biases, presenting results that suggest these can be no more pronounced when shadowing political elites than in other forms of observational research.

When making causal inferences, post-treatment confounders complicate analyses of time-varying treatment effects. Conditioning on these variables naively to estimate marginal effects may inappropriately block causal pathways and may induce spurious associations between treatment and the outcome, leading to bias. To avoid such bias, researchers often use marginal structural models (MSMs) with inverse probability weighting (IPW). However, IPW requires models for the conditional distributions of treatment and is highly sensitive to their misspecification. Moreover, IPW is relatively inefficient, susceptible to finite-sample bias, and difficult to use with continuous treatments. We introduce an alternative method of constructing weights for MSMs, which we call “residual balancing”. In contrast to IPW, it requires modeling the conditional means of the post-treatment confounders rather than the conditional distributions of treatment, and it is therefore easier to use with continuous treatments. Numeric simulations suggest that residual balancing is both more efficient and more robust to model misspecification than IPW and its variants in a variety of scenarios. We illustrate the method by estimating (a) the cumulative effect of negative advertising on election outcomes and (b) the controlled direct effect of shared democracy on public support for war. Open-source software is available for implementing the proposed method.

Declining telephone response rates have forced several transformations in survey methodology, including cell phone supplements, nonprobability sampling, and increased reliance on model-based inferences. At the same time, advances in statistical methods and vast amounts of new data sources suggest that new methods can combat some of these problems. We focus on one type of data source—voter registration databases—and show how they can improve inferences from political surveys. These databases allow survey methodologists to leverage political variables, such as party registration and past voting behavior, at a large scale and free of overreporting bias or endogeneity between survey responses. We develop a general process to take advantage of this data, which is illustrated through an example where we use multilevel regression and poststratification to produce vote choice estimates for the 2012 presidential election, projecting those estimates to 195 million registered voters in a postelection context. Our inferences are stable and reasonable down to demographic subgroups within small geographies and even down to the county or congressional district level. They can be used to supplement exit polls, which have become increasingly problematic and are not available in all geographies. We discuss problems, limitations, and open areas of research.

Supervised machine learning methods are increasingly employed in political science. Such models require costly manual labeling of documents. In this paper, we introduce active learning, a framework in which data to be labeled by human coders are not chosen at random but rather targeted in such a way that the required amount of data to train a machine learning model can be minimized. We study the benefits of active learning using text data examples. We perform simulation studies that illustrate conditions where active learning can reduce the cost of labeling text data. We perform these simulations on three corpora that vary in size, document length, and domain. We find that in cases where the document class of interest is not balanced, researchers can label a fraction of the documents one would need using random sampling (or “passive” learning) to achieve equally performing classifiers. We further investigate how varying levels of intercoder reliability affect the active learning procedures and find that even with low reliability, active learning performs more efficiently than does random sampling.

Researchers employing qualitative comparative analysis (QCA) and its variants use two-element Boolean algebra to compare cases and identify putative causal conditions. I show that the two-element Boolean algebra constrains research in three important ways: it restricts what we can say about sets and the interactions between sets, it embodies a logical language that is too weak to capture modern social science theories, and it restricts our analysis of causation to necessity and sufficiency accounts and does not allow for counterfactuals. Modern quantitative analysis suffers none of these restrictions and provides a much richer way to understand the social world.