OP20 Is The Quality Of Evidence In Health Technology Assessment Deteriorating Over Time?: A Case Study On Cancer Drugs In Australia

Abstract

Introduction

Recently, there have been concerns regarding a trend toward poorer quality evidence being accepted by regulatory institutions and the consequent impact on health technology assessment (HTA) decision-making. This study aimed to determine whether there has been a change in the quality of evidence provided on cancer drugs proposed for listing on the Pharmaceutical Benefits Scheme, using data solely extracted from public summary documents (PSD) published by the Australian government.

Methods

PSDs published from July 2005–2020 were reviewed. Metrics associated with quality of evidence were extracted, including the directness of comparison, study design, sample size, and risk of bias (RoB). Additional data were extracted to provide greater context to any observed trends in quality of evidence. Analyses were performed across different time periods. Associations between the quality of evidence and time periods were explored using logistic regression analysis.

Results

In total, 214 PSDs were included in the analysis. Only 13 percent of submissions provided a single arm study or observational study as the key evidence; however, 37 percent of submissions did not contain a direct (‘head-to-head’) comparison relevant to Pharmaceutical Benefits Advisory Committee (PBAC) decision-making. Among all submissions containing direct evidence, about half had findings of a moderate/high/unclear RoB. Among all submissions containing indirect comparisons, over half had transitivity issues. In submissions containing direct comparisons, there was an increase in the RoB over time even after adjusting for trial data maturity and the rareness of the drug indication (odds ratio [OR] 1.30; 95% confidence interval [CI] 0.99, 1.70). There were no clear time trends observed in sample size, directness, study design, or transitivity issues during any of the observed time periods.

Conclusions

In the last 7 years, a high proportion of cancer drug submissions presented findings with a high RoB and transitivity issues. As the evidence dossiers provided to the PBAC are often congruent with submissions made elsewhere, this poor evidence quality is of concern and can only lead to higher levels of decision-maker uncertainty.