The validity of conclusions drawn from specific research studies must be evaluated in light of the purposes for which the research was undertaken. We distinguish four general types of research: description and point estimation, correlation and prediction, causal inference, and explanation. For causal and explanatory research, internal validity is critical – the extent to which a causal relationship can be inferred from the results of variation in the independent and dependent variables of an experiment. Random assignment is discussed as the key to avoiding threats to internal validity. Internal validity is distinguished from construct validity (the relationship between a theoretical construct and the methods used to operationalize that concept) and external validity (the extent to which the results of a research study can be generalized to other contexts). Construct validity is discussed in terms of multiple operations and discriminant and convergent validity assessment. External validity is discussed in terms of replicability, robustness, and relevance of specific research findings.

This chapter focuses on different research designs in experimental sociology. Most definitions of what constitutes an experiment converge on the idea that the experimenter "control" the phenomenon under investigation, thereby setting the conditions under which the phenomenon is observed and analyzed. Typically, the researcher exerts experimental control by creating two situations that are virtually identical, except for one element that the researcher introduces or manipulates in only one of the situations. The purpose of this exercise is to observe the effects of such manipulation by comparing it with the outcomes of the situation in which the manipulation is absent. One way to look at how the implementation of this rather straightforward exercise produces a variety of designs is by focusing on the relationship that experimental design bears with the theory that inspires it. Therefore, we begin this chapter with a discussion of the relationship between theory and experimental design before turning to a description of the most important features of various types of designs. The chapter closes with a short overview of experiments in different settings such as laboratory, field, and multifactorial survey experiments.

Sociology is a science concerning itself with the interpretive understanding of social action and thereby with a causal explanation of its course and consequences. Empirically, a key goal is to find relations between variables. This is often done using naturally occurring data, survey data, or in-depth interviews. With such data, the challenge is to establish whether a relation between variables is causal or merely a correlation. One approach is to address the causality issue by applying proper statistical or econometric techniques, which is possible under certain conditions for some research questions. Alternatively, one can generate new data with experimental control in a laboratory or the field. It is precisely through this control via randomization and the manipulation of the causal factors of interest that the experimental method ensures – with a high degree of confidence – tests of causal explanations. In this chapter, the canonical approach to causality in randomized experiments (the Neyman–Rubin causal model) is first introduced. This model formalizes the idea of causality using the "potential outcomes" or "counterfactual" approach. The chapter then discusses the limits of the counterfactual approach and the key role of theory in establishing causal explanations in experimental sociology.

In the introduction, the field of experimental sociology is outlined and the core concepts of manipulation and control, as well as two crucial conditions of control, are introduced. The random allocation of participants to the treatment and the control group ensures that exogenous factors are distributed equally across these groups, which allows to evaluate the effect of the manipulated condition. Incentivization helps operationalizing behavioral assumptions into the experimental condition. The chapter then briefly elaborates on the topics of the following chapters.

Laboratory experiments are the type of study that most people have in mind when talking about experiments. In this chapter, we first discuss the strengths of laboratory experiments, which offer the highest degree of experimental control as compared to other types of experiments. Single factors can be manipulated according to the requirements of theories under highly controlled conditions. As such, laboratory experiments are well-placed to test theories. We then introduce a sociological laboratory experiment as a leading example, which we use as a reference for a discussion of several principles of laboratory research. Furthermore, we discuss a second goal of laboratory experiments, which is the establishment of empirical regularities in situations where theory does not provide sufficient guidance for deriving behavioral expectations. The chapter concludes with a short discussion of caveats for the analysis of sociological data generated in laboratory experiments.

The first sociological experiments have been conducted in the second and third decades of the twentieth century, accompanied by a fierce debate about the possibilities and limits of the approach, which anticipated many of the critiques currently raised against the method. The chapter traces the development of experimental research in sociology from these beginning to modern perspectives. One of the reasons for the marginal position of experimentation in sociology has been the reluctance to give up full control of potentially intervening variables (called the ex post facto method) in favor of randomization. Inspirations from social psychology and, later, economics, have finally resulted in the experimental designs that are currently used in sociology.

Chapter 5 moves from theory into evidence and discusses how empirical economists think about causality. First, the chapter covers common issues that make it difficult to have confidence in causal claims based on associational evidence alone. Then, experimental evidence is discussed: how to run an experiment, common pitfalls that can undermine confidence in experimental evidence, and what can be done to avoid them. Next, major experimental studies on the impact of health insurance are described. Finally, the chapter discusses the concept of quasi-experimental evidence and how it fits into economics. The end of chapter supplement discusses ethics in research with human subjects and the role of institutional review boards.

Confronting models with data is only effective when the statistical model matches the biological one and the structure of your data collection is right for the statistical model. We outline some basic principles of sampling, emphasizing the importance of randomization. Randomization is also essential to experimental design, but so are controls, replication of experimental units, and independence of experimental units. This chapter emphasizes the distinction between sampling or experimental units representing independent instances and observational units representing things we measure or count from those units. Observational units may be subsamples of experimental units, but shouldn’t be confused with them. In this chapter, we also introduce methods for deciding how much data you need.

Bias means systematic error. Its most common form is confounding bias, where various factors in the context of treatment influence the results, without the awareness of clinician or patient. Incorrect claims are made when these confounding factors are ignored. Randomization is the best solution to confounding bias. Clinical examples are provided for antidepressant discontinuation in bipolar depression and for the relationship between substance abuse and antidepressant-related mania. Other forms of bias are discussed, such as measurement bias.

Randomization solves the problem of confounding bias; it addresses systematic error, which is the most important source of error, not chance. It equalizes all potential confounding factors, known and unknown, in all groups so that they equally influence the results, and thus can be ignored. Only then can the results of randomized treatment be interpreted at face value and causal inferences made. Sample size and other factors are relevant, though, and small randomized clinical trials (RCTs) can be misleading. Examples are given.

The basic insight of evidence-based medicine is that randomized studies are more valid in their results than observational studies or case studies or clinical experience, in that order, because of correction for confounding bias. This concept of levels of evidence is the key to understanding EBM.

There is a case to be made for evidence-based medicine (EBM), and there is a case to be made against it. Many of the critiques of EBM are ill-founded, but some important criticisms need attention. The issues and concerns around EBM are discussed.

The two major sources of error are chance (random error) and confounding bias (systematic error). After correcting for these two kinds of error, one can then assess or assert causation. These are the “Three Cs.”

The topic of clinical trials is introduced using the example of the MRC trial in streptomycin in TB. The role of randomization, the subject of design of experiments and ethical problems in conducting trials in patients are covered.

This chapter presents the case for employing Bayesianism as a universal, unified framework for inference that narrows the divide between qualitative versus quantitative data, within-case versus cross-case analysis, and observational versus experimental research. It offers a Bayesian critique of various other approaches to qualitative methods and multi-method research.