6 - Significance testing and effect size estimation

Summary

Once we have defined the variables to be investigated in our research, perhaps first redefining them into composites as described in the last chapter, we can begin to analyze “the results” of our research. Too often in the behavioral sciences, however, the results are equated with a series of tests of significance that may be only tangentially related to the questions that motivated the research in the first place. Two reasons why tests of significance are not informative enough for them to serve as the definition of the results of our research are:

1. a. They give no indication of the magnitude of the effect under investigation, and

2. b. They are often based on more than a single df in the numerator of an F test or on more than a single df for a χ² test. In both cases a significant result alone does not tell us how a specific variable X is related to a specific variable Y.

It is more useful to think of the results as the answer to the question: What is the relationship between any variable X and any variable Y? (Rosenthal, 1984). The variables X and Y are chosen with only the constraint that their relationship be of interest to us.