Hostname: page-component-78c5997874-v9fdk Total loading time: 0 Render date: 2024-11-16T14:31:24.241Z Has data issue: false hasContentIssue false
  • English
  • Français

Inadequate reporting of trials compromises the applicability of systematic reviews

Published online by Cambridge University Press:  21 July 2009

Gerald Gartlehner ,
Patricia Thieda ,
Richard A. Hansen ,
Laura C. Morgan ,
Janelle A. Shumate  and
Daniel B. Nissman
Gerald Gartlehner
Affiliation:
Danube University, Krems
Patricia Thieda
Affiliation:
University of North Carolina at Chapel Hill
Richard A. Hansen
Affiliation:
University of North Carolina at Chapel Hill
Laura C. Morgan
Affiliation:
University of North Carolina at Chapel Hill
Janelle A. Shumate
Affiliation:
University of North Carolina Hospitals
Daniel B. Nissman
Affiliation:
Medical University of South Carolina
Get access Rights & Permissions [Opens in a new window]

Abstract

Background: Uncertainty about the applicability of controlled trial findings is an increasing concern for clinicians and policy decision makers. This study aimed to determine whether information reported in studies included in systematic reviews was adequate enough to assess their applicability.

Methods: We used the databases of four recently conducted systematic reviews on the comparative efficacy and safety of second-generation antidepressants, inhaled corticosteroids, Alzheimer's drugs, and targeted immune modulators. We developed and pilot-tested a questionnaire to assess the adequacy of reporting with respect to seven previously validated criteria of study design that distinguish explanatory from pragmatic studies. For each of the 137 included studies, two reviewers independently assessed the adequacy of reporting.

Results: Overall, only 12 percent of the included studies provided sufficient information to reliably distinguish explanatory from pragmatic studies. The areas with the greatest lack of reporting were the setting of the study, methods of adverse event assessment, and sample size considerations to determine a minimally important difference from a patient perspective.

Conclusions: Substantial shortcomings in reporting exist in aspects of study design important to determine whether a study is applicable to specific populations of interest.

Keywords

EffectivenessEfficacyPragmatic studiesExplanatory studiesSystematic reviewsMethods study
Type
General Essays
Information
International Journal of Technology Assessment in Health Care , Volume 25 , Issue 3 , July 2009 , pp. 323 - 330
Copyright
Copyright © Cambridge University Press 2009

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

1. Agency for Healthcare Research and Quality (AHRQ). http://effectivehealthcare.ahrq.gov/aboutUs/index.cfm (accessed November 15, 2008).Google Scholar
2. Altman, DG, Schulz, KF, Moher, D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. 2001;134:663–694.Google Scholar
3. Atkins, D. Creating and synthesizing evidence with decision makers in mind: integrating evidence from clinical trials and other study designs. Med Care. 2007;45 (Suppl 2):S16–S22.Google Scholar
4. Chan, AW, Hrobjartsson, A, Haahr, MT, et al. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291:2457–2465.Google Scholar
5. Dumville, JC, Torgerson, DJ, Hewitt, CE. Reporting attrition in randomized controlled trials. BMJ. 2006;332:969–971.Google Scholar
6. Ethgen, M, Boutron, I, Baron, G, et al. Reporting of harm in randomized, controlled trials of nonpharmacologic treatment for rheumatic disease. Ann Intern Med. 2005;143:20–25.Google Scholar
7. Fortin, M, Dionne, J, Pinho, G, et al. Randomized controlled trials: do they have external validity for patients with multiple comorbidities? Ann Fam Med. 2006;4:104–108.Google Scholar
8. Gartlehner, G, Hansen, RA, Kahwati, L, et al. Drug class review on second generation antidepressants. http://www.ohsu.edu/ohsuedu/research/policycenter/customcf/derp/product/AD2_Final_Report_%20Update%203.pdf (accessed November 15, 2008).Google Scholar
9. Gartlehner, G, Hansen, RA, Nissman, D, et al. A simple and valid tool distinguished efficacy from effectiveness studies. J Clin Epidemiol. 2006;59:1040–1048.Google Scholar
10. Gartlehner, G, Hansen, RA, Thieda, P, et al. Drug class review on targeted immune modulators. http://www.ohsu.edu/ohsuedu/research/policycenter/customcf/derp/product/TIM_Final_Report_Update%2012.pdf (accessed November 15, 2008).Google Scholar
11. Godwin, M, Ruhland, L, Casson, I, et al. Pragmatic controlled clinical trials in primary care: the struggle between external and internal validity. BMC Med Res Methodol. 2003;3:28.Google Scholar
12. Gross, CP, Mallory, R, Heiat, A, et al. Reporting the recruitment process in clinical trials: who are these patients and how did they get there? Ann Intern Med. 2002;137:10–16.CrossRefGoogle Scholar
13. Hansen, RA, Gartlehner, G, Kaufer, DJ, et al. Drug class review on Alzheimer's drugs. http://www.ohsu.edu/ohsuedu/research/policycenter/customcf/derp/product/ALZ_Final_Report_Update%201.pdf (accessed November 15, 2008).Google Scholar
14. Hansen, RA, Gartlehner, G, Lohr, KN, et al. Drug class review on inhaled corticosteroids. http://www.ohsu.edu/ohsuedu/research/policycenter/customcf/derp/product/ICS_Final_Report_Update%201.pdf (accessed November 15, 2008).Google Scholar
15. Haynes, B. Can it work? Does it work? Is it worth it? The testing of healthcare interventions is evolving. BMJ. 1999;319:652–653.Google Scholar
16. Hazell, L, Shakir, SA. Under-reporting of adverse drug reactions: a systematic review. Drug Saf. 2006;29:385–396.Google Scholar
17. Hewitt, C, Hahn, S, Torgerson, DJ, et al. Adequacy and reporting of allocation concealment: review of recent trials published in four general medical journals. BMJ. 2005;330:1057–1058.Google Scholar
18. Huwiler-Muntener, K, Juni, P, Junker, C, et al. Quality of reporting of randomized trials as a measure of methodological quality. JAMA. 2002;287:2801–2804.Google Scholar
19. Ioannidis, JP, Evans, SJ, Gotzsche, PC, et al. Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med. 2004;141:781–788.Google Scholar
20. MacRae, KD. Pragmatic versus explanatory trials. Int J Technol Assess Health Care. 1989;5:333–339.Google Scholar
21. Malmivaara, A, Koes, BW, Bouter, LM, et al. Applicability and clinical relevance of results in randomized controlled trials: the Cochrane review on exercise therapy for low back pain as an example. Spine. 2006;31:1405–1409.Google Scholar
22. Moher, D, Cook, DJ, Eastwood, S, et al. Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement. Onkologie. 2000;23:597–602.Google Scholar
23. Moher, D, Schulz, KF, Altman, D. The CONSORT Statement: Revised recommendations for improving the quality of reports of parallel-group randomized trials 2001. Explore (NY). 2005;1:40–45.Google Scholar
24. Mullner, M, Matthews, H, Altman, DG. Reporting on statistical methods to adjust for confounding: a cross-sectional survey. Ann Intern Med. 2002;136:122–126.Google Scholar
25. Mulrow, CD. Rationale for systematic reviews. BMJ. 1994;309:597–599.Google Scholar
26. Oregon Health and Science University (OHSU). http://www.ohsu.edu/drugeffectiveness/description/index.htm (accessed November 15, 2008).Google Scholar
27. Plint, AC, Moher, D, Morrison, A, et al. Does the CONSORT checklist improve the quality of reports of randomized controlled trials? A systematic review. Med J Aust. 2006;185:263–267.Google Scholar
28. Schwartz, D, Lellouch, J. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis. 1967;20:637–648.Google Scholar
29. Slutsky, J, Atkins, D, Chang, S, et al. Comparing medical interventions: AHRQ and the effective health-care program. J Clin Epidemiol. 2008 [Epub ahead of print].Google Scholar
30. Sokka, T, Pincus, T. Eligibility of patients in routine care for major clinical trials of anti-tumor necrosis factor alpha agents in rheumatoid arthritis. Arthritis Rheum. 2003;48:313–318.Google Scholar
31. Zwarenstein, M, Treweek, S, Gagnier, JJ, et al. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 2008;337:a2390.Google Scholar