Is the assumption of waning of treatment effect applied consistently across NICE technology appraisals? A case-study focusing on disease-modifying therapies for treatment of multiple sclerosis

Abstract

Objectives

Whether the effects of therapies may wane over time is a matter of debate, especially when considering their long-term cost-effectiveness. Here, we examined how the assumption of the waning of treatment effect was applied across the National Institute for Health and Care Excellence (NICE) appraisals for disease-modifying therapies (DMTs) used in multiple sclerosis.

Methods

We undertook a document analysis following a search of the NICE website. The inclusion criteria of the study were as follows: all publicly available documents related to completed appraisals for DMTs (period: January 2000 to July 2021). The exclusion criteria of the study were as follows: all documents that did not meet the inclusion criteria, especially pertaining to drugs used in other disease areas. We extracted information about the waning of treatment effect assumption as considered by companies, assessment groups, and appraisal committees, and we analyzed trends over time.

Results

We reviewed fifteen appraisals that reported guidance on sixteen DMTs. Irrespective of the drugs’ mechanism of action or their pharmaceutical nature, there was substantial variation in the modalities when the assumption of waning was implemented. We noted the recent preference to use all-cause discontinuation as a proxy. This heterogeneity did not appear to affect acceptance of the DMTs (all but one were recommended for use across the National Health System (NHS)).

Conclusions

Modeling the long-term effect of therapies is challenging, especially given the limited follow-up duration of related trials. This generates recurrent debates on the presence of waning of treatment efficacy and heterogeneity across appraisals. More refined recommendations obtained by consensus among stakeholders could help to achieve greater consistency in decision making.

Introduction

Multiple sclerosis (MS) is a chronic demyelinating disease of the central nervous system with typical onset between the ages of 20 and 40 years. MS affects approximately 2.5 million people globally, with one in every 500 people in the UK living with MS (1). People living with MS are at an increased risk of mortality compared to the general population, and may also experience broader impacts on their lifestyle, health, and well-being. MS not only impacts the individual’s health-related quality of life but also that of their caregivers.

The availability of disease-modifying therapies (DMTs) has led to an evolving scene in patient management. DMTs decrease the frequency of relapses and their associated disability and may also delay underlying disease progression. Over the last 20 years, a broad spectrum of DMTs has become available, ranging from the earlier beta-interferon and glatiramer acetate to the latest generation of drugs, such as oral agents (dimethyl fumarate, teriflunomide) and humanized monoclonal antibodies. Several DMTs have been recommended in the UK for people with different forms of MS (relapsing–remitting (RRMS), primary progressive (PPMS), and secondary progressive (SPMS). All but one of these have been recommended as cost-effective for use in defined subpopulations by the National Institute for Health and Care Excellence (NICE) through its technology appraisal process (2).

NICE relies on economic modeling to aid in decision making about how to allocate healthcare resources within a constrained budget. The purpose of modeling is to structure clinical and economic evidence in a form that can be used to inform decisions about clinical practices and the allocation of limited resources to achieve maximum benefits for the population (3). As part of the technology appraisal process, drug manufacturers present submissions with clinical and economic evidence to support and demonstrate the clinical and cost-effectiveness of their intervention compared to other recommended treatments. This evidence is critically appraised by an independent external assessment group, which provides an unbiased report to NICE, highlighting concerns (2).

One such concern is the assumption about whether the treatment effect may wane over the time horizon of the economic models. This issue is frequently debated during NICE technology appraisal committee meetings, especially when appraising DMTs. Waning of treatment benefit may exist while on treatment, but in the context of this work, we consider waning of the treatment effect as any decrease in benefit during continued treatment and/or reduction in sustained or differential benefit following discontinuation of treatment when supposed benefit is extended beyond the observation period. Economic models usually extend well beyond the time when treatment with DMTs has ceased and well beyond randomized-controlled trial (RCT) and observational data collection periods. Implementing an assumption of waning of the treatment effect is therefore lacking in evidence; however, it may have substantial impacts on the accrual of both costs and quality-adjusted life years (QALYs), and therefore on the key metric for decision making, the incremental cost-effectiveness ratio (ICER) for a new drug when compared to current therapies. A clear need exists to support either inclusion or exclusion of treatment waning of DMTs in future economic analyses where clear evidence does not exist. To understand the application of waning assumptions in real-world policy decision making and consider future recommendations, we reviewed all committee papers (technology appraisals/assessments and external assessment group reports) of DMTs recommended by NICE, and to summarize whether and how waning of the treatment effect has been included in the model-based economic analyses. We summarize the impact of this assumption on decision making across NICE appraisals of DMTs.

Methods

We conducted a document analysis to review all model-based cost-effectiveness analyses of DMTs appraised by NICE, which was undertaken based on a search on the NICE website.

There was no prior conceptual framework to this research work which was initiated by the two lead authors (XA and PA) based on their experience with appraisals for DMTs.

Three reviewers (XW-S, XA, and PA) screened documents that met the following inclusion criteria: all committee papers, appraisal consultation documents (ACDs), final appraisal documentation (FAD), and guidance for DMTs appraised by NICE for the treatment of people living with RRMS, PPMS, or SPMS, undertaken from January 2000 up to July 2021.

The exclusion criteria of the study were all documents that did not meet the inclusion criteria, especially those related to drugs used in other disease areas, or those on DMTs that were ongoing at the time the searched ended (July 2021).

We scrutinized the references included in the company’s submissions and assessment groups’ reports and contacted experts in the field to ensure the completeness of the information. Given the nature of this review, we have not searched common electronic databases, as we envisaged that all committee papers/submissions would be stored on the NICE website. Similarly, we did not undertake any grey literature searching since, as far as we are aware, the process set up by NICE ensures comprehensive and transparent publication of technical documents used for decision making (4). The reference lists from submissions/evidence review group (ERG) reports were checked for previous technology appraisals.

Study selection strategy and data extraction

Three reviewers (XW-S, XA, and PA) screened the titles of all committee papers. Any disagreements between the reviewers were resolved by discussion or by recourse to a fourth reviewer (MC). The study flow and reasons for the exclusion of submissions are documented in the flow diagram (Figure 1).

Figure. 1. Flow diagram.

We have not undertaken a formal reporting/methodological quality assessment because the economic analyses reported in the companies’ submissions have been appraised by external assessment groups and NICE appraisal committees. Data were extracted using a prepiloted data extraction form, by one reviewer and cross-checked for accuracy by a second reviewer. Any disagreements were resolved by discussion or by recourse to a third reviewer. We extracted data on whether the company included the assumption of waning of the treatment effect in their economic analysis and, if so, how it was implemented. Additionally, we extracted whether the assumption had been included in the external assessment groups analyses and their justification for this. We also extracted the NICE appraisal committees’ preference along with their justification for dealing with the assumption of waning of the treatment effect of DMTs.

Reviewers extracted relevant information from the committee’s papers, including the application of the treatment effect (confirmed disability progression (CDP) and annualized relapse rate), if waning of the treatment effect had been included in the analysis (if so, in the base-case and/or scenario analysis), and waning applied to the treatment effect (CDP and/or annualized relapse rate).

Any disagreements between reviewers were resolved by discussion of recourse to a fourth reviewer (MC).

Results

The search of the NICE website identified 134 records covering guidance, NICE advice, news, and research recommendations. One additional relevant record was identified through other sources. Based on titles and brief summaries, 120 records were excluded, with fifteen records (5–19) considered relevant. Full details of the selection process and reasons for exclusion are presented in Figure 1. We identified thirteen single-technology appraisals (STAs) (5–7;9–15;17–19) and two multiple technology assessment (8;16) reporting guidance on 16 DMTs for the treatment of people living with MS in the UK. Details of the documents available for each appraisal and or assessment are reported in Table 1.

Table 1. Characteristics of the disease-modifying therapies submitted to NICE for appraisal

CDP, confirmed disability progression; DMT, disease-modifying therapies; EAG, evidence assessment group; ERG, evidence review group; FAD, final appraisal documentation; MS, multiple sclerosis; PPMS, primary progressive multiple sclerosis; RRMS, relapsing–remitting multiple sclerosis; SPMS, secondary progressive multiple sclerosis; STA, single-technology appraisal.

Across the three types of MS, we identified twelve appraisals undertaken for the treatment of people living with RRMS only (5–8;12–15;17–20) and one for RRMS and SPMS, (16) PPMS only (10) and SPMS only (11). Fifteen DMTs were recommended and one DMT was not recommended (18). One DMT (15) was subsequently withdrawn by the company due to safety concerns. Nine DMTs were categorized as biological drugs and seven as nonbiological drugs. Of the DMTs categorized as a biological drug, six were humanized monoclonal antibody drugs and the remaining three were recombinant proteins (beta-interferons).

The characteristics of the company submissions are summarized in Table 1.

All companies undertook an economic evaluation using a Markov model to determine the cost-effectiveness of DMTs for treating adults living with MS. All submissions reported cost-effectiveness in terms of cost per QALY.

Companies’ approaches to waning of the treatment effect

Consistent with the NICE reference case, all economic analyses were undertaken over a lifetime horizon using annual cycle lengths, and all undertook their analyses from the NHS and personal social services perspective. Nine submissions (5–7;9;13–15;17;18) included an assumption of waning of the treatment effect in their economic analyses. Four of those (5;6;17;18) included this assumption in their base-case analysis and in scenario analyses. Five (7;9;13–15) included the assumption in their scenario analyses only. Two studies (11;19) stated that treatment discontinuation could be considered as a proxy for waning of the treatment effect. Four (8;12;16;21) submissions did not consider waning and/or specifically stated that waning was not a plausible consideration or was not supported by evidence.

All companies that included the assumption of a waning of the treatment effect in their original submission provided a rationale for its inclusion. The most reported justifications were that this assumption represented a conservative approach or that waning was adopted to be in line with previous appraisals. Companies that had not included the assumption of waning of the treatment effect in their original economic analyses stated that there was no evidence from their clinical trials or from follow-up studies that the treatment effect of DMTs is likely to wane over time.

Table 1 shows that the majority of the company submissions implemented the assumption by applying a stepped reduction to the treatment effect (disability progression either confirmed at 3 or 6 months) for disease progression (for example, TA624 applied the same reduction (Years 1–2: no waning, years 3–5: 25 percent reduction and years 6+: 50 percent reduction) across all DMTs included in the economic analyses). We also noted that several approaches as shown in Figure 1 were used by the companies to implement waning of the treatment effect. It was assumed in TA656 (11) that treatment discontinuation could be used as a proxy for any reduction in treatment effectiveness. We are unaware of any company submission that measured/reported treatment discontinuation due to a perceived loss of treatment effect; many potential reasons for treatment discontinuation exist, including disease progression, perceived lack of effect, adverse events, and rules about treatment duration. For those companies that included an assumption of waning, we sought to extract information to explore the impact of this assumption on the ICERs. However, we were unable to record this information from all appraisals as some of the results had been redacted or the analysis was not undertaken (8–11;15;18;19).

External assessment group approaches to waning of the treatment effect

External assessment groups provided an independent critique of the companies’ submissions as part of the NICE technology appraisal process. We found that five different assessment groups out of the nine assessment groups have critically assessed all DMTs submitted to NICE (12;16). The external assessment groups agreed that waning of the treatment effect was a concern and included it in their base-case and scenario analyses. In the one submission that did not include the assumption of waning of the treatment effect, the external assessment group agreed with the company’s approach that all-cause discontinuation could be used as a proxy for waning of the treatment effect (11). External assessment groups justified including the assumption in their base-case analysis in various ways including consistency with waning of the treatment effect implemented in previous appraisals; (9–11;17;18) as a precaution due to the lack of evidence (9;10;13;14;17–19) from long-term studies that could support an assumption that the treatment effect would be sustained or that the effect does not wane over time; concordance with clinical expert opinion or in accordance with previously expressed appraisal committee’s preference (for RSS) (8).

Appraisal committees’ approaches to waning of the treatment effect

Fourteen appraisals/assessments (5–11;13–19) were reviewed by the NICE Appraisal Committee B and one (12) was appraised when the Appraisal Committee was undifferentiated. In 11 appraisals (5–8;10;11;13–15;17;18), the Committee preferred including the assumption of waning of the treatment effect in their list of preferred assumptions ahead of decision making. Of the remaining four appraisals/assessments, we were unsure in one appraisal (16) if waning of the treatment effect was discussed, waning of the treatment effect was not discussed in one appraisal (12) and in two appraisals (9;19), the Appraisal Committee preferred that all-cause discontinuation should be used as a proxy for treatment waning. Figure 2 summarizes the committees’ preferences for implementing an assumption of waning of the treatment effect over time. The Appraisal Committees usually provided justification for including or excluding waning of the treatment effect in their lists of preferred assumptions. Examples of reasons for including waning of the treatment effect encompassed: based on clinical expert opinion (8;15) lack of evidence about the long-term impact of DMTs (5;15); a cautious approach (14); treatment discontinuation rates could be used a proxy (9); and in line with previous appraisals (11).

Fig. 2. Key dates, with the modality of implementing the assumption of waning of the treatment effect in DMTs appraised by NICE.

Discussion

Economic models should be built on the best available evidence, but there will be instances where evidence is not available or is too short-term to capture important elements that might be included in an economic model. Analysts rely on expert opinion for building their economic models or make assumptions to extrapolate beyond the available evidence. Decision-making agencies such as NICE, suggest that the costs and benefits of a health technology should be demonstrated over a lifetime horizon commensurate with the likely duration of all major costs and consequences (2). Several methods are available to extrapolate beyond the observed trial data (22). These methods all have different functional forms that could potentially result in different estimates of the treatment effect beyond the observed data (22). The NICE Decision Support Unit Technical Support Document sets out suggestions for best practices for choosing an appropriate method to extrapolate; however, there remains some inherent uncertainty in method selection that can have an impact on cost-effectiveness estimates. Methods of sensitivity and scenario analyses are available that can help address this uncertainty and should be used consistently in the appraisal of health technologies.

We recognize that the concept of waning of treatment effect is not explicitly defined. For example, the NICE methods guide indicates that “alternative scenarios should also be routinely considered to compare the implications of different methods for extrapolation of the results” (2). And more specifically with regards to the duration of treatment effects for which different scenarios might be included with “the treatment benefit in the extrapolated phase being either nil, the same as during the treatment phase and continuing at the same level, or diminishing in the long term” (2).

Interestingly, this statement was present in the first version of the NICE methods guide published in 2004. However, our work indicates that waning of the treatment effect was considered only from the second STA of a DMT, namely that of fingolimod (14), whereas it was not considered as a concern in the first STA conducted for natalizumab (12). Based upon our search of NICE technology appraisal guidance published prior to 2012, we hypothesize that the fingolimod appraisal was one of the first, if not the first, appraisal to incorporate considerations of waning of treatment effect, although only as a sensitivity analysis. Of note, this appears to have been spontaneously initiated by the drug manufacturer. We cannot speculate on whether the concept would have been introduced anyway in later appraisals; however, the fingolimod appraisal presumably opened a new era of MS STAs where waning of treatment effect became systematically discussed not only for DMTs but also for technologies beyond the scope of MS.

Our review of health technologies of DMTs over time reveals other landmark decision points. The first one corresponds to the teriflunomide appraisal (13) for which the NICE appraisal committee chose to apply, among its preferred assumptions, a waning of treatment effect in the base-case model (increasing the CDP hazard ratio (HR) by 50 percent after 5 years), whereas the company considered it as a scenario analysis in its original submission. The second landmark pertains to the appraisal of ocrelizumab used for RRMS, (9) for which the NICE Appraisal Committee agreed, that as suggested by the company in its submission, treatment discontinuation could be considered as a proxy for treatment effect waning (no change of CDP HR over time was applied). This decision provided a precedent for companies to apply treatment discontinuation as a marker of loss of effectiveness in subsequent appraisals for biological (ocrelizumab used in PPMS, ofatumumab) or nonbiological (siponimod) drugs. Although the NICE Appraisal Committee reiterated its position for the two former drugs, it invalidated the company’s preferred assumption for the latter on the grounds that the “conventional” modality of treatment effect waning (i.e., through the increase of the CDP HR from a certain time point) should be applied for siponimod consistent with assumptions made for a drug of the same class, namely fingolimod.

In our view, adopting intratherapeutic class consistency is desirable and should be recommended. However, the scientific, clinical, or pharmacologic rationale to adopt treatment discontinuation as a proxy for waning for these biological drugs, as opposed to a conventional treatment effect waning for nonbiological drugs such as siponimod or teriflunomide, is unsupported by evidence. This may also call for a greater level of intertherapeutic class consistency in the manner of considering treatment waning taking into account factors such as the duration of drug effect or the drug mechanism of action. In circumstances where there is a small-time period between treatment discontinuation and end of observation relative to the life-time horizon, and where at end of observation a large proportion of patients remain at risk of the outcome event, then long-term extrapolation of the treatment effect size would be difficult to justify; this would seem especially the case where reasons for treatment discontinuation (particularly waning of effectiveness) are unspecified.

We are aware of a third method of implementing the assumption of a waning of treatment effect, which was proposed by the External assessment groups within the scope of the appraisal for ocrelizumab used for PPMS. It consists of combining a reduction of treatment effect from a certain time point (by increasing the CDP HR) with an increase of treatment discontinuation for the assessed drug from the same time point. The rationale is that should a waning of treatment effect exist, diminishing the relative effect of a new drug over the other, this would result in an increased perceived loss of effectiveness by patients, which would lead to an increased rate of discontinuation with the new treatment.

With regards to biological drugs, we have noted that some companies have attempted to justify the absence of a consideration of waning of treatment effect in their base-case arguing that monoclonal antibodies were partially or fully humanized, which is thought to reduce the risk of neutralizing antibodies formation targeting the drugs. However, the link between neutralizing antibodies formation and reduced treatment effectiveness is not well documented and does not get around the fact that the main treatment effect is on reducing relapses but as time proceeds main cause of progressive disability is secondary progression which is probably less affected by all DMTs.

Although many DMTs used within the scope of MS have been appraised by the same Appraisal committee at NICE, we have found variation in how waning of the treatment effect had been applied. Indeed, the increase of the CDP HR could occur either gradually over the lifetime horizon or could vary from different time points (after 2, 5, or 11 years). Consistency in application of waning should be made by those tasked with making these important policy decisions.

The variations on time points used in the consideration of the waning of treatment effect should presumably reflect any available evidence produced by companies about the sustained effectiveness of their drug based on long-term follow-up studies carried out as part of open-label extensions of RCTs. Although these studies can be relevant to assess the long-term outcomes and safety of DMTs, our view however is that they provide very limited evidence about whether the effect of DMTs relative to the comparator used in randomized component of the trial is maintained over time.

Limitations

This work presents some limitations. First, we focused our review on DMTs used for the treatment of MS, which means that our observations cannot be extended beyond this scope. Although DMTs represent a minority of drugs among those appraised through the technology appraisal program, we assume that this study represents an exemplary case on the difficulty of considering the concept of waning of treatment effect in STAs in general, including for anticancer therapies. Second, we conducted our analyses using published documents available on the NICE website. For example, in this work, we were unable to accurately document the impact of applying an assumption of waning of the treatment effect because most of the analyses, whether conducted with a waning effect applied in the base-case or sensitive analyses, were available only with redacted ICERs and therefore not in the public domain. However, based on our reading of appraisal documents, we did not find a DMT which was not recommended by NICE on the grounds that it was not cost-effective to the NHS after applying a waning of treatment effect. Third, we also aimed to assess the impact of applying a waning of treatment effect in terms of decision making, which proved to be a challenge. Applying a waning effect by increasing the CDP HR should result in reducing the incremental effectiveness, hence increasing the ICER. Conversely, with treatment discontinuation applied as a proxy for waning, the cumulated costs associated with a new therapy can be reduced which can lead to reduced incremental costs, hence reduced ICER. To review the practical impact on decision making, we had planned to examine whether applying a waning of treatment effect reduced /increased the ICER below/above the £30,000 per QALY threshold but this was not possible.

It is obvious that our quantitative analyses could have been complemented by qualitative analyses, by means of interviews of main stakeholders (NICE Appraisal Committee members, companies, and ERGs), and use of thematic analysis (23), which could be relevant to better capture the full complexity of the topic.

Strengths

To the best of our knowledge, this is the first study to review how the concept of waning of the treatment effect has been applied as part of a major health technology assessment program. Considering our review, more explicit recommendations could be considered as part of the NICE methods guide. Such recommendations could originate by consensus from a panel of stakeholders including members of the NICE technology appraisal teams, clinicians, ERG and Health Technology Assessment (HTA) body members, companies, and patients’ representatives.

As part of future research, this work could be extended beyond the scope of MS and into more precise estimates of the effects of treatment waning on the cost-effectiveness of drugs and on relevant ICERs. Similarly, given that the waning of the treatment effect appears to be predominantly a concept, a more fundamental question can be raised, which is whether a waning of the treatment effect for DMTs has ever been evidenced.

Other methodological approaches such as the use of a qualitative framework (e.g., thematic analysis) could be considered in the next steps of research.

Last, the perspective of the work could be broadened to examine whether the assumption of waning of treatment effect is also discussed and applied consistently by other HTA agencies such as The Canadian Agency for Drugs & Technologies in Health (CADTH), the French National Authority for Health (HAS), or the Agency for Healthcare Research and Quality (AHRQ). This could be the starting point that could justify the need to elaborate on international HTA guidance on the handling of waning in the assessment of health technologies.

Conclusion

Uncertainty in the long-term effect of DMTs presents a challenge when estimating their cost-effectiveness over a lifetime horizon. Waning of the treatment benefit may exist while patients with MS are receiving DMTs. An assumption of waning has been proposed by UK decision makers when considering the cost-effectiveness of DMTs, but there is inconsistency on how this assumption is implemented across technology appraisals over time. The waning assumption, how it is implemented and at which timepoint may have an impact on the cost-effectiveness results, and by corollary, the decisions made. More refined recommendations should be considered in methods guides, as treatment waning in DMTs is only one part of the much-needed effort to address how waning is captured in long-term economic analyses as it is likely to exist in other drugs used to treat other diseases/conditions.