4 - Field Experiments with Survey Outcomes

Summary

Field experiments with survey outcomes are experiments where outcomes are measured by surveys but treatments are delivered by a separate mechanism in the real world, such as by mailers, door-to-door canvasses, phone calls, or online ads. Such experiments combine the realism of field experimentation with the ability to measure psychological and cognitive processes that play a key role in theories throughout the social sciences. However, common designs for such experiments are often prohibitively expensive and vulnerable to bias. In this chapter, we review how four methodological practices currently uncommon in such experiments can dramatically reduce costs and improve the accuracy of experimental results when at least two are used in combination: (1) online surveys recruited from a defined sampling frame (2) with at least one baseline wave prior to treatment (3) with multiple items combined into an index to measure outcomes and, (4) when possible, a placebo control for the purpose of identifying which subjects can be treated. We provide a general and extensible framework that allows researchers to determine the most efficient mix of these practices in diverse applications. We conclude by discussing limitations and potential extensions.