Hostname: page-component-5f745c7db-nc56l Total loading time: 0 Render date: 2025-01-06T07:11:56.977Z Has data issue: true hasContentIssue false
  • English
  • Français

A General Linear Model for Estimating Effect Size in the Presence of Publication Bias

Published online by Cambridge University Press:  01 January 2025

Jack L. Vevea  and
Larry V. Hedges
Jack L. Vevea*
Affiliation:
University of Chicago
Larry V. Hedges
Affiliation:
University of Chicago
*
Requests for reprints and for the computer program employed in this work may be directed to Jack L. Vevea, University of North Carolina at Chapel Hill, NC 27599-3270.
Get access Rights & Permissions [Opens in a new window]

Abstract

When the process of publication favors studies with small p-values, and hence large effect estimates, combined estimates from many studies may be biased. This paper describes a model for estimation of effect size when there is selection based on one-tailed p-values. The model employs the method of maximum likelihood in the context of a mixed (fixed and random) effects general linear model for effect sizes. It offers a test for the presence of publication bias, and corrected estimates of the parameters of the linear model for effect magnitude. The model is illustrated using a well-known data set on the benefits of psychotherapy.

Keywords

meta-analysisresearch synthesispublication biaseffect sizemixed modelsselection models
Type
Original Paper
Information
Psychometrika , Volume 60 , Issue 3 , September 1995 , pp. 419 - 435
Copyright
Copyright © 1995 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Authors' note: The contributions of the authors are considered equal, and the order of authorship was chosen to be reverse-alphabetical.

References

Begg, C. B. (1994). Publication bias. In Cooper, H., Hedges, L. V. (Eds.), The handbook of research synthesis (pp. 399–409). New York: Russell Sage Foundation.Google Scholar
Begg, C. B., Berlin, J. A. (1988). Publication bias: A problem in interpreting medical data (with discussion). Journal of the Royal Statistical Society, Series A, 151, 419–463.CrossRefGoogle Scholar
Bozarth, J. D., Roberts, R. R. (1972). Signifying significant significance. American Psychologist, 27, 774–775.CrossRefGoogle Scholar
Cooper, H., Hedges, L. V. (1994). The handbook of research synthesis, New York: Russell Sage Foundation.Google Scholar
Coursol, A., Wagner, E. E. (1986). Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias. Professional Psychology, 17, 136–137.CrossRefGoogle Scholar
Dawes, R. M., Landman, J., Williams, M. (1984). Discussion on meta-analysis and selective publication bias. American Psychologist, 39, 75–78.Google Scholar
Dear, K. B. G., Begg, C. B. (1992). An approach for assessing publication bias prior to performing a meta-analysis. Statistical Science, 7, 237–245.CrossRefGoogle Scholar
Dickersin, K., Min, Y-I, Meinert, C. L. (1991). The fate of controlled trials funded by the NIH in 1979. Controlled Clinical Trials, 12, 634–634.CrossRefGoogle Scholar
Dickersin, K., Min, Y-I, Meinert, C. L. (1992). Factors influencing the publication of research results: Followup of applications submitted to two institutional review boards. Journal of the American Medical Association, 267, 374–378.CrossRefGoogle ScholarPubMed
Easterbrook, P. J., Berlin, J. A., Gopalan, R., Matthews, D. R. (1991). Publication bias in clinical research. Lancet, 337, 867–872.CrossRefGoogle ScholarPubMed
Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1–20.CrossRefGoogle Scholar
Hedges, L. V. (1984). Estimation of effect size under nonrandom sampling: The effects of censoring studies yielding statistically insignificant mean differences. Journal of Educational Statistics, 9, 61–85.CrossRefGoogle Scholar
Hedges, L. V. (1992). Modeling publication selection effects in meta-analysis. Statistical Science, 7, 246–255.CrossRefGoogle Scholar
Hedges, L. V., Olkin, I. (1985). Statistical methods for meta-analysis, New York: Academic Press.Google Scholar
Hedges, L. V., & Vevea, J. L. (1993). Estimating effect size under publication bias: Small sample properties and robustness of a selection model. Manuscript submitted for publication.Google Scholar
Iyengar, S., Greenhouse, J. B. (1988). Selection models and the file drawer problem. Statistical Science, 3, 109–135.Google Scholar
Kendall, M., Stuart, A. (1979). The advanced theory of statistics. Volume 2, Inference and relationship 4th ed.,, London and High Wycombe: Charles Griffin and Company.Google Scholar
Lane, D. M., Dunlap, W. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107–112.CrossRefGoogle Scholar
Light, R. J., Pillemer, D. B. (1984). Summing up: The science of reviewing research, Cambridge, MA: Harvard University Press.CrossRefGoogle Scholar
Melton, A. W. (1962). Editorial. Journal of Experimental Psychology, 64, 553–557.CrossRefGoogle Scholar
National Research Council (1992). Combining information: Statistical issues and research opportunities, Washington, DC: National Academy Press.Google Scholar
Nelson, N., Rosenthal, R., Rosnow, R. L. (1986). Interpretation of significance levels by psychological researchers. American Psychologist, 41, 1299–1301.CrossRefGoogle Scholar
Rosenthal, R., Gaito, J. (1963). The interpretation of levels of significance by psychological researchers. Journal of Psychology, 55, 33–38.CrossRefGoogle Scholar
Rosenthal, R., Gaito, J. (1964). Further evidence for the cliff effect in the interpretation of levels of significance. Psychological Reports, 4, 570–570.CrossRefGoogle Scholar
Smith, M. L. (1980). Publication bias in meta-analysis. Evaluation in Education, 4, 22–24.CrossRefGoogle Scholar
Smith, M. L., Glass, G. V., Miller, T. I. (1980). The benefits of psychotherapy, Baltimore: The Johns Hopkins University Press.Google Scholar
Sterling, T. C. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance or vice versa. Journal of the American Statistical Association, 54, 30–34.Google Scholar
Vevea, J. L., Clements, N. C., Hedges, L. V. (1993). Assessing the effects of selection bias on validity data for the general aptitude test battery. Journal of Applied Psychology, 78, 981–987.CrossRefGoogle Scholar
White, K. R. (1982). The relation between socioeconomic status and achievement. Psychological Bulletin, 31, 461–481.CrossRefGoogle Scholar