10 - Comparing and combining judgment studies

Summary

Even after a thorough and well-done analysis of the results of our judgment study, we are not quite finished. We will need to put the results of our study into the context of other studies addressing the same questions. We will want to know whether our results are consistent with, or different from, the results of other studies; those conducted by us and those conducted by others. In this chapter we present some of the basic quantitative procedures for comparing our results with other results and for combining our results with other results. The procedures that will be described, the so-called meta-analytic procedures, will prove valuable even when we have not conducted any judgment studies of our own. Any review of the literature will profit from the application of meta-analytic procedures, a fact becoming rapidly clear to investigators wanting to review bodies of literature. In 1976, for example, there were six meta-analytic publications; in 1982 there were 120 (Lamb & Whitla, 1983). The early 1980s also saw the emergence of a series of textbooks describing in varying degrees of detail the history, theory, problems, and procedures of meta-analysis (Cooper, 1984; Glass, McGaw, & Smith, 1981; Hedges & Olkin, 1985; Hunter, Schmidt, & Jackson, 1982; Light & Pillemer, 1984; Mullen & Rosenthal, 1985; Rosenthal, 1980, 1984).