Industry Funding by Itself is Not a Reason for Rating Down Studies for Risk of Bias

To evaluate how study characteristics and methodological aspects compare based on presence or absence of industry funding, Hughes et al. conducted a systematic survey of randomized controlled trials (RCTs) published in three major medical journals. The authors found industry-funded RCTs were more likely to be blinded, post results on a clinical trials registration database (ClinicalTrials.gov), and accrue high citation counts.¹ Conversely, industry-funded trials had smaller sample sizes and more frequently used placebo as the comparator, used a surrogate as their primary outcome, and had positive results.

Some individuals and teams conducting systematic reviews believe that industry funding per se always puts such trials at high risk of bias and that one should rate down certainty in them accordingly. We believe this is misguided. Indeed, industry-sponsored trials are typically far better funded than investigator-initiated RCTs, allowing much greater scrutiny regarding issues that include concealment of randomization, integrity of blinding procedures, adherence to protocol, and implementing measures to minimize loss to follow-up.

Industry sponsors typically utilize expensive contract research organizations that allow far more detailed oversight of procedures at individual centers than trials funded through public agencies. This is particularly true of RCTs conducted to achieve regulatory approval in which industry sponsors are aware of the high level of inquiry that regulatory agencies are likely to implement.

These considerations suggest that industry-funded trials should fare equally well or superiorly in mitigating risk of bias relative to investigator-initiated RCTs. Indeed, this is the case. A prior systematic survey comparing industry-funded and non-industry-funded studies demonstrated similar performance in sequence generation, allocation concealment, follow‐up and selective outcome reporting. Further, the study demonstrated that industry-funded trials are more often protected against bias through blinding procedures.² A second survey of RCTs of drug therapies for rheumatoid arthritis found that industry-funded trials were more frequently blinded, provided an adequate description of participant flow, and incorporated an intention-to-treat analysis.³ These two surveys substantiate the inference that resource and oversight considerations will, in general, result in industry-funded studies doing as well or better on several aspects of methodological rigor than investigator-funded studies.

However, numerous evidence syntheses, including the linked study by Hughes and colleagues, have found that industry-sponsored studies are associated with disproportionately positive findings relative to studies funded by other sources.⁴ If it isn’t risk of bias, what explains the phenomenon?

Vested intellectual and financial interests pose the largest threat to the trustworthiness of industry-funded studies. For-profit organizations may be inherently prone to designing and interpreting the findings of a study involving their therapeutic intervention overly optimistically and over-emphasizing the importance of their findings. Indeed, the evidence suggests that this is very much the case: industry-funded studies are more likely to be enthusiastic about treatments under investigation.⁵

Numerous strategies may produce this phenomenon, referred to as “spin,” and lead to mischaracterized or misleading results. These include inappropriate interpretations of results for a given study design (e.g. interpreting non-significant results as being “equally good” in a superiority trial), inappropriate extrapolations unsupported by results, selective reporting (including omission of non-significant outcomes and over-emphasis on significant but less important surrogate outcomes and secondary analyses), misleading or over-favorable data presentation, undermining certainty in results, and shifting framing of the abstract or conclusions to a different objective.⁶ Moreover, some methodological features can also favor investigated interventions. This includes the use of suboptimal comparators (including placebo where therapies with proven efficacy exist or suboptimal active controls)⁷ and composite outcomes (where outcomes with variable patient importance, incidence and treatment effects are combined).⁸ Meta-epidemiological studies have consistently demonstrated that these design and interpretation features, which extend beyond standard risk of bias criteria, consistently lead to disproportionately favorable results in industry-funded trials relative to their non-industry-funded counterparts, and overly sanguine interpretations of results when drawing conclusions.⁹

Vested intellectual and financial interests pose the largest threat to the trustworthiness of industry-funded studies. For-profit organizations may be inherently prone to designing and interpreting the findings of a study involving their therapeutic intervention overly optimistically and over-emphasizing the importance of their findings.

Such inappropriate conclusions have led — perhaps understandably — to claims that evidence-based medicine has been hijacked to serve the agendas of conflicted parties, including for-profit organizations, rather than primarily focusing on scientific inquiry.¹⁰ Indeed, evidence-based medicine must defend the integrity of the research that provides the basis for clinical decision-making. Fortunately, leaders in evidence-based medicine have worked hard to do so.¹¹

Publication bias represents a second mechanism of evidence distortion whereby study results influence their likelihood of being published. Studies with positive or statistically significant findings are more likely to be published than their “negative study” counterparts, leading to overestimated treatment effects and threatening the validity and overall certainty in a body of evidence.¹² Selective publication has been evident in industry-sponsored research for over two decades.¹³

When faced with an industry-funded trial (or any study in which vested intellectual or financial interests may be present), readers — clinicians, patients, and fellow researchers — should maintain a healthy skepticism to avoid being led astray by misleading claims and biased inferences.¹⁴ In addition to considering the methodological quality based on traditional risk of bias criteria,¹⁵ readers should: (1) focus on the methods and results of studies to guide their interpretations rather than relying on the author’s interpretation presented in the discussion; (2) beware of faulty comparators and composite end-points; (3) exercise caution when interpreting small treatment effects and subgroup analyses¹⁶; and (4) ascertain the extent to which spin may influence results and, when considering studies together, to which positive treatment effects may be over-represented (or negative or non-significant effects under-represented). Alternatively, pre-appraised evidence resources such as the ACP Journal Club,¹⁷ trustworthy practice guidelines such as the BMJ Rapid Recommendations series¹⁸ and other evidence-based point-of-care clinical resources such as UpToDate ¹⁹ and DynaMed ²⁰ offer balanced and methodologically sound interpretations of published studies.²¹ Such resources may be particularly helpful to those with no training on health research methodology.