The purpose of this chapter is to give readers a sense of the breadth of experimental applications in the social sciences. The chapter reviews lab, field, and survey experiments, as well as naturally-occurring experiments such as lotteries. Each type of experiment is illustrated by reviewing in detail an exemplary study, drawing from experimental literature in psychology, development economics, health, and political science. Special attention is paid to the design choices that researchers made when recruiting subjects, measuring outcomes, and allocating subjects to experimental conditions. Discussion of each study includes the analysis of its main statistical findings. By showing how experiments are designed and analyzed, this chapter lays the groundwork for the practice experiment that readers will undertake in Chapter 6.

Here we address bias and causality, beginning with the bias against failure in the existing science of science research. Because the data available to us is mostly on published papers, we necessarily disregard the role that failure plays in a scientific career. This could be framed as a surviorship bias, where the “surviving” papers are those that make it to publication. This same issue can be seen as a flaw in our current definition of impact, since our use of citation counts keeps a focus on success in the discipline. We explore the drawbacks and upsides of variants on citation counts, including altmetrics like page views. We also look at how possible ways to expand the science of science to include unobservable factors, as we saw in the case of the credibility revolution in economics. Using randomized controlled trials and natural experiments, the science of science could explore causality more deeply. Given the tension between certainty and generalizability, both experimental and observational insights are important to our understanding of how science works.