20 - Statistical tools for meta-analysis: From straightforward to esoteric

Summary

Introduction

A traditional meta-analysis is an analytical synthesis of the literature on a research topic, which leads to summary judgments about the overall significance and typical sizes of effects reported in that research domain. A more idealized conceptualization of meta-analysis views the objective to be the estimation of an effect-size surface, which gives the expected effect sizes in the research area as a function of scientifically relevant factors. The effect-size surface perspective is more difficult to implement successfully in a data analysis than the literature-synthesis perspective because the effect-size surface can be estimated only by extrapolating a response surface estimated from observed studies to an idealized region of perfect studies.

Simple and direct statistical tools are particularly valuable for straightforward analyses supporting the literature-synthesis perspective. Technically complicated statistical tools that attempt to adjust for selection bias in the published versus unpublished studies in a literature synthesis are not really appropriate because the conceptualization of a population of studies from which existing studies are drawn is scientifically of little relevance. In contrast, the estimands of the effect-size surface perspective are scientifically relevant, and therefore it is appropriate to develop and use whatever tools are needed to draw inferences for these estimands. Continued work is needed on both fronts: straightforward statistical tools that can be used in literature synthesis and appropriate statistical tools for effect-size surface estimation, even if esoteric.

This chapter explains and advances this thesis by reviewing, first, the traditional view of meta-analysis and indicating, by illustration, the continuing need for the development of direct statistical tools for literature synthesis; and, second, by criticizing this perspective as valuable but scientifically limited, and proposing and arguing for the superiority of the effect-size surface perspective.