Considering and communicating uncertainty in health technology assessment

Abstract

As health technology assessment (HTA) seeks to combine complex sets of evidence, values, and perspectives to support open, accountable, and transparent decision making, uncertainty is inherent. Uncertainty is present in the clinical and economic inputs that inform HTA and is a critical factor during context-specific deliberations where the evidence is weighed and decisions are made, taking uncertainty into account through either financial or evidence-generation mechanisms. The presence and impact of uncertainty must also be communicated to all relevant stakeholders during the HTA output stage. This article summarizes the 2021 HTAi Global Policy Forum discussion on “Considering and Communicating Uncertainty in HTA” that debated some of the key challenges and opportunities regarding uncertainty in HTA. Through a combination of small and large group discussions, core themes related to the topic of uncertainty in HTA were identified. These discussions revealed that: utilization of a life cycle/HTA management approach helps manage uncertainty; genuine stakeholder input and engagement (and not just consultation) can clarify uncertainty; tolerance of risk, the relationship of risk to uncertainty, and the context in which uncertainty is considered is critical; transparent and early dialogues could be increased to further reduce the uncertainty during HTA; and communicating uncertainty in HTA outputs is critical. The paper ends with suggested next steps that HTA agencies and stakeholders (such as industry, patients, regulators, payers, and others) might take to move the field forward. The paper promotes further discussion on aspects of uncertainty that should be more openly discussed, debated, and addressed.

Introduction

Health technology assessment (HTA) is defined as “a multidisciplinary process that uses explicit methods to determine the value of a health technology at different points in its life cycle. The purpose is to inform decision-making in order to promote an equitable, efficient, and high-quality health system” (1). Although the concept of uncertainty is not specifically defined in the multiorganizational HTA Glossary (2), the idea is clearly articulated in many of the definitions and activities pertaining to HTA. By the very nature of what HTA is and what it seeks to do, uncertainty will always exist at some level. Therefore, considering uncertainty is a fundamental and inherent component of all elements of HTA.

The processes and methods for considering uncertainty are context- and situation-specific. “Contexts” that differ include jurisdictions with differing legal views pertaining to uncertainty; HTA settings/remits that vary globally, possibly resulting in various relationships between an HTA body, the health system, and payers; and societal values that might change the acceptability of uncertainty that overlies the settings in which the HTA agencies operate. “Situations” can apply to considering uncertainty in the current pandemic setting (e.g., urgent, “rule of rescue” care) or in the cases of rare or very severe disease. In some countries, situations in which technologies are considered for children or disadvantaged populations may also result in additional tolerance for greater uncertainty (3) (with tolerance of uncertainty reflecting how much uncertainty a deliberative committee is willing to accept). This is all affected by the appetite for risk within an HTA organization generally as well as for the individuals deliberating on its behalf (i.e., how risk averse or risk seeking a deliberative committee is willing to be, with more uncertainty likely to be acceptable for more risk-seeking bodies). Some uncertainty, however, will always remain (given the nature of HTA as described), and there is a judgment to be made about acceptable levels of uncertainty regarding evidence of benefits, harms, and costs that will influence findings and policies. These risks are not static, as they may increase with a greater likelihood of treatment benefit proving to be very different than expected, either in a positive direction (where a wrong decision denies access to a beneficial treatment) or in a negative one (funding a technology that proves ineffective).

Considering uncertainty consistently across settings is, therefore, challenging due to these contextual dependencies. The impact on the decision (i.e., the willingness of an HTA agency to accept a higher level of risk and decide with an uncertain evidence base) is a key consideration for all stakeholders, particularly patients, the public, technology manufacturers, and policy makers/payers.

These were the primary drivers for choosing this complex, multifactorial issue as the topic for the 18th HTAi Global Policy Forum (GPF), held virtually on 22, 24, and 26 February 2021 (https://htai.org/policy-forum/global-policy-forum/). The meeting was attended by ninety-nine representatives from not-for-profit organizations (public HTA agencies, private HTA organizations, payers, and health system representatives) and for-profit organizations (pharmaceutical, biotech, and device companies), five patient representatives, invited speakers, and HTAi leadership (Supplementary Table 1). The objective of the meeting was to review and discuss both challenges and opportunities in considering and communicating uncertainty in HTA, to learn about innovations and novel approaches in this space, and to discuss supporting actions that HTA bodies and stakeholders might take. To frame the GPF, the perspective taken was that of HTA agencies (rather than, e.g., of the regulator or payer), with discussions aimed at a strategic, policy level.

This article presents a summary of the discussions held at the 2021 GPF meeting. The meeting was conducted under the Chatham House Rule (4), whereby participants are free to share information obtained at the meeting, but they may not reveal the identity or affiliation of the person providing the information. Our summary presents the authors’ view on the proceedings and is not a consensus or official statement from individuals who attended the meeting or of their organizations. However, all members of the GPF had an opportunity to comment on the paper prior to publication. This summary is intended, in part, as an invitation to others involved in HTA to discuss and debate the issues raised.

Meeting Structure and Proceedings

To inform the meeting discussion and activities, a Background Paper was developed (5), which presented an overview of key methods and considerations related to considering and communicating uncertainty that are currently available in the published literature and are commonly used by GPF member organizations. To help facilitate and structure the discussion about considering and communicating uncertainty in HTA, an “input-throughput-output” (ITO) model was used (Figure 1) (6;7). In this context, the ITO framework describes uncertainty in terms of:

1. (1) Input: the collection of information, evidence and perspectives with an exploration of the presence and impact of uncertainty that inform HTA activities. The types of input uncertainty primarily considered by deliberative bodies in HTA include clinical uncertainty; economic model structural uncertainty; economic model parameter uncertainty and affordability uncertainty.

2. (2) Throughput: the critical examination of evidence and weighting of the uncertainty and its impact. This is the stage where the interplay between the types of input uncertainty is considered and a decision (either by consensus or through voting) is sought. Importantly, handling uncertainty during deliberations will vary according to perspectives and values, contexts, and situations, and the risk tolerance of individual deliberators will vary, which can, in turn, be affected by how uncertainty is presented (i.e., “deliberative” uncertainty). Uncertainty is typically managed at the throughput stage either financially (with a greater level of uncertainty usually being more acceptable if a lower price is paid or risk-sharing agreements are struck), through more evidence generation, or for certain populations or diseases increasing the tolerance to uncertainty.

3. (3) Output: the communication of the outcome, the level of uncertainty, its impact and any actions (such as recommendations for more evidence generation) are consolidated at the output stage. The importance of clearly conveying the presence and impact of uncertainty and statistical concepts and methods to explore uncertainty in an understandable way to interested stakeholders cannot be underestimated. Stakeholders may include patients, the public, the media, clinicians, technology manufacturers, researchers, and the health system, including policy makers and payers.

Figure 1. ITO framework for considering and communicating uncertainty in HTA.

For each stage, there are opportunities within the process where stakeholders (such as patients, clinicians, manufacturers, and payer representatives) can contribute and participate. This crosses all of the ITO domains and is a fundamental component when considering and communicating uncertainty in HTA with stakeholders able to provide (for example) clarity around input uncertainty, provide context at the throughput stage, and ensure feedback to agencies at the output stage.

The GPF itself was also structured over three daily 2.5-hour sessions using the ITO framework. The input domain was the focus of presentations and plenary discussions on Day 1, the throughput domain the focus of perspective and case study presentations and breakout discussions on Day 2, and the output domain the focus of patient-focused presentations, a panel session, and breakout discussions on Day 3. For the breakout group discussions on Days 2 and 3, the GPF members were divided into six groups of roughly equal size that were composed of similar numbers of for-profit and not-for-profit members, with an aim of having one patient representative attend per group. Each breakout group was assigned a discussion facilitator and a rapporteur who was responsible for taking notes and summarizing the discussion. Breakout questions are included in Table 1.

Table 1. Breakout questions

The keynote presentations, case studies, panels, breakouts, and plenary discussions provided members with the opportunity to think about the importance of various activities pertaining to considering and communicating uncertainty in HTA from a variety of stakeholder perspectives (HTA agencies, regulators, patient organizations, industry, and payers). Live polling using Slido (www.sli.do.com) was employed as a real-time tool to gather feedback on the key “take-home” messages during the final session.

Meeting Products

The ITO model proved helpful in framing the discussion about a complex concept and the discussions of each day are presented accordingly below. Although we recognize that industry and other HTA stakeholders contend with process uncertainties, such as changes in project scope or modeling approach, or inconsistencies in evidentiary judgments across appraisal committees, the primary focus of the discussions was on the treatment of uncertainty within the ITO framework. There were, however, several overarching themes that transcended this framework, and these are also summarized as core themes.

Input

A prerecorded keynote presentation was available before the first session that considered the topic of “Fighting Misinformation in an Era of Uncertainty,” using the COVID-19 pandemic as a backdrop. This meant that the GPF began with the concept of trust and honesty in communication. Live discussion at the session touched on the importance of accurate representation of the totality of evidence (with avoidance of “expert creep” where unqualified individuals discuss subjects outside of their expertise) and the need to be genuine with the public about what is known and what is not known. This is critical in increasing public engagement and trust in scientific communications, which occur in a system with inherent knowledge-generation pressures caused by condensing nuanced reports into media briefings and other summaries (8).

The session included presentations representing the regulatory and manufacturer perspectives, focusing on how they interact with the uncertainty HTA receives and any shared responsibility for mitigating it. The following key points related to input uncertainty were highlighted during member discussions:

1. (1) Uncertainty will always exist in HTA; this is due to the nature of what HTA is and there are different needs served in comparison with the regulatory environment, where the primary concern is that of the risks and benefits of a technology in terms of safety and efficacy.

2. (2) Uncertainty exists on a spectrum rather than being linear or binary; approaches such as cost-effectiveness analysis and orientation toward an acceptable cost-effectiveness output (i.e., “the number”) may give the perception of less uncertainty.

3. (3) Education of all stakeholders about all aspects of uncertainty is important; for example, the notion that uncertainty is not always predictable is an important consideration for all stakeholders to comprehend.

4. (4) Uncertainty is context- and situation-specific; considering uncertainty consistently across settings and topics is, therefore, challenging due to contextual dependencies.

Regarding types of input uncertainty, most uncertainty in HTA likely arises because regulators are increasingly willing to approve technologies on different types and more limited evidence (in terms of, e.g., sample size, length of follow-up, or use of controlled study design and no data on costs, potential offsets and impacts important to patients), and there is limited time to address these data and study design concerns before an HTA is conducted (noting that many countries are moving to HTA at the time of product launch to increase the potential for timely patient access). A key challenge is the limited opportunities for regulators and HTA agencies to have early precompetitive collaborations to discuss and align, where feasible, on evidence expectations prior to the registration of a final trial design.

The use of real-world evidence (RWE) was also discussed, as both a method for generating evidence for initial HTA consideration and an approach for managing risk-sharing arrangements post adoption. However, as discussed at a previous GPF, this requires the development and enforcement of quality standards. If care is not taken, RWE can itself increase rather than mitigate uncertainty (9). Other challenges include the possibility of either an uncertainty pricing “discount” or certainty “bonus” at the time of approval—this is currently being explored in some jurisdictions and could help mitigate the impact of uncertainty from a financial perspective. Finally, the “opportunity costs” associated with uncertainty are a key concept to consider (10); acknowledging that this is bidirectional (i.e., the contribution of uncertainty to decisions to fund therapies that do not live up to their potential and to reject treatments that ultimately prove their value) and exists at both the investment and disinvestment ends of the technology life cycle.

Throughput

The second session included presentations on the Summary of Information for Patients (The Summary) and a project being coordinated by the HTAi Patients and Citizens Interest Group (PCIG, https://htai.org/interest-groups/pcig/projects/current-projects/) to expand the use of the Summary throughout multiple jurisdictions. This highlighted the benefit of iterative dialogue between HTA agencies, industry, and patient groups to provide appropriate and targeted input and context about the patient experience. It may also be used to resolve areas where greater transparency would be beneficial. The importance of understanding where patient input can have the greatest impact on reducing uncertainty for deliberative committees (e.g., with “commentaries” around the effects of rare diseases) was discussed. Areas such as these were acknowledged as presenting particular challenges, given that patient experiences can vary substantially and can be highly personal. However, the role that patient groups may play in providing registry data, obtaining multiple perspectives from individual patients, and identifying key uncertainties in the patient experience points to the need for more robust patient-based evidence. There are increasing examples of evaluation of the committee experiences with patient inputs and reflections on how types of evidence are documented in the final recommendations; this is an important area of further research (11).

Case studies from HTA agencies followed with approaches to characterize uncertainty for deliberators and improvements and innovations that are being considered. These presentations highlighted:

1. (1) important methodological updates (such as the increasing use of probabilistic sensitivity and value of information analyses);

2. (2) more clarity around certain contexts in which greater uncertainty may be acceptable (such as for technologies for rare diseases, innovative technologies, or those that provide a large benefit); and

3. (3) techniques for consistently characterizing and presenting uncertainty to deliberators (e.g., visualizations, color-coded systems, or new methods adapting the Grading of Recommendations Assessment, Development and Evaluation [GRADE] approach to document and convey the influence of uncertainty and other determinative factors using standardized descriptors).

After these presentations, members reflected that all approaches may require some adaptation dependent on context, particularly in the cases of advanced therapeutics and rare diseases (e.g., precision oncology therapies). Relying on the use of managed entry agreements (MEAs) or price reductions to manage uncertainty may be insufficient, and other approaches (e.g., a greater use of RWE, more innovative trial designs, more opportunities for regulators and HTA agencies to engage and align on registration trial design, evidence packages, and a more tolerant approach to risk if/when appropriate) could be considered. In addition, the presentation to, and understanding of, uncertainty by a committee is one challenge, but this still needs to be transferred in a transparent and understandable way to decision makers.

Members highlighted the reality that different agencies have different approaches to managing uncertainty, and this can create challenges for stakeholders (particularly for manufacturers). Consistency and predictability in how a committee considers and manages uncertainty (including acceptance of advanced methodologies for managing uncertainty and the dynamics surrounding the committee deliberations themselves) is a key consideration, with predictability and early signaling around technology considerations of particular importance for manufacturers as an aid for designing evidence packages.

Output

The third session included presentations from the patient perspective and a case study from a patient-centered organization. Both presentations highlighted the communication of uncertainty from the patient and public/citizen perspectives and the importance of explaining the presence, impact of, and any efforts to reduce uncertainty for these stakeholder groups. Creating health literate materials for patients and the public (but also other stakeholders such as decision- and policy makers, politicians, and the media) was highlighted as of critical importance (12). As an example, relative and absolute risks are used variably (seemingly dependent on whether the effects are positive or negative), and contingent probabilities can be difficult to understand for stakeholders, who often resonate best to the presentation of risks or outcomes in natural frequency terms. The importance of context was also reiterated; for example, a 1 in 100 chance of success can provide the value of hope to a patient (with this often not considered in HTA currently) but is unlikely to be viewed as positive from the perspective of the payer who must fund treatment for all 100.

A reaction panel reflected that payers are facing challenges related to uncertainty every day, as they may be required to make reimbursement decisions even when HTA recommendations differ in scope or are absent entirely. Where there are differences in remit and perspectives, this can also create a disconnect between patient and payer needs regarding outcomes, which can make it challenging for the manufacturers to know what to prioritize. Stakeholders look for a number of elements when assessing how uncertainty is presented in the context of HTA outputs, for example: consistency of assessment of the same evidence across jurisdictions; continuous and iterative processes so that uncertainty can be addressed, and outputs updated; and transparency about the role of uncertainty and trust in the decision-making framework. It was highlighted that the consequences of uncertainty and the future actions to be taken to address uncertainties must be clearly described to all stakeholders in an understandable way (i.e., accounting for health literacy standards (13)). Inclusion of the totality of evidence (including qualitative data, patient-centered outcomes, and other stakeholder inputs) in communications about uncertainty is critical.

To better present and convey the uncertainty to deliberative committees, clinical and economic data could be presented separately (rather than as a single incremental cost-effectiveness ratio) to help determine their independent contributions to overall uncertainty. Gains and losses in health that relate to uncertainty in the data could be presented visually, and summary representations of uncertainty may be useful initial tools with detailed exploration of the most impactful uncertainties to follow (with a suggestion that resolving all uncertainties is not possible and a pragmatic approach is required). Given the complexities of the methods used to characterize and explore the impact of uncertainty, as mentioned, health literacy is an important factor to consider. Importantly, this concept applies not only to patients, but also to all stakeholders involved in the HTA process, from the deliberators/committee members through to clinicians, decision makers, and payers who need to fully understand the presence and impact of uncertainty. Learnings from other industries (e.g., aviation's use of visualizations, lay instructions, a common language, and empowerment of other staff to check the pilot's actions) may be possible, and in certain instances, employing mediators may be a useful way to help determine the resolvability of uncertainty.

Investment in communications expertise is required so that messaging is done at an appropriate level; and there are resource implications for ensuring that communication is done effectively to multiple audiences. Innovations in communication are, therefore, important (such as storytelling, use of infographics, and visual summaries) and methods to prioritize explanation of the uncertainties that are key to a decision (rather than explaining all the uncertainties). Acknowledging the patient perspective and tolerance to risk is also important. The notion that uncertainty may not necessarily be a “bad” thing (e.g., in the context of a breakthrough innovative treatment that is by nature more uncertain with an immature evidence base, but one that brings the value of hope or possibility of living long enough to receive further advancements in treatment (14)) should not be ignored. Explaining the concept of opportunity costs, alongside conveying the humanity of deliberations to all stakeholders involved in HTA is critical to inform, build trust, and increase understanding. Ensuring public trust in decision making by making HTA inputs and outputs more accessible and acceptable by using transparent and understandable processes could even increase compliance and lead to better treatment effects.

Core Themes

Although the sessions were structured using the ITO framework, there were themes that recurred throughout the discussions that cross the boundaries between input, throughput, and output.

Life Cycle/HTA Management Approach

An HTA life cycle/HTA management approach (i.e., from pre market, during market approval, post market, through to the disinvestment of a technology (1)) was considered by members as a potentially efficient way to manage the uncertainty that is inherent throughout the HTA process. However, although more early advice/dialogue, life cycle management, evidence generation post recommendation, monitoring MEA, and updating HTA reviews would be desirable, these are all incredibly resource-intensive activities. To do this fully requires investment from all stakeholders (particularly HTA agencies), and so some actions must be prioritized over others. How this prioritization is done must be communicated in a similar fashion to how overall uncertainty should be communicated—that is, clearly, transparently with multistakeholder participation and engagement (not just consultation). HTA could also even be made “multi-phase,” with traditional clinical research addressing safety and efficacy first, followed by longer-term collection of RWE that is weighted appropriately and addresses more patient-centered “life-goals” (i.e., those measures of impact that are most meaningful to patients) (15).

Stakeholder Input and Engagement

HTA agencies working more closely with key stakeholders (such as patients) was noted as crucial to help deliberative committees better understand the lived experience, but also to receive context and nuance around treatment benefits and risks that may not be evident from a purely quantitative, scientific review of the available data. Stakeholders such as clinicians (with deep knowledge of practice and clinical pathways) and patients (with input possibly scored or semiquantified using core outcome sets) can play a greater role to provide additional context around uncertainty. Evidence in this form could be derived from rapid studies (e.g., surveys) to address uncertainties in close to real time. Ensuring that external stakeholders have sufficient time to reflect on and prepare this evidence was noted as an area of improvement for many HTA agencies. It was also highlighted that this increased collaboration with stakeholders should be across the HTA life cycle and should continue post appraisal. Finally, understanding the difference between uncertainty in the inputs (data uncertainty) and those that might be introduced by the deliberation itself (such as differences in opinion, different attitudes and tolerance of risk, uncertainty in the implementation or diffusion of a technology) was noted as important.

Tolerance of Risk

As mentioned previously, uncertainty is highly context-dependent and tolerance of uncertainty (or conversely, appetite for risk) will vary by stakeholder groups, even down to the individual level (16). Factors that influence tolerance for uncertainty include what the subsequent effects of a decision might be (e.g., the likelihood of further high-quality research, or the possibility to effectively change or rescind a recommendation). The perspective and tolerance of uncertainty between HTA agencies and regulators is also continuing to diverge, with accelerated approvals and consequent limitations in evidence, and so now it may be the time to revisit harmonization efforts between these bodies and pilot projects (with possible expansion across multiple countries) (17). Such activities may also help HTA agencies and stakeholders focus efforts on identifying the resolvable uncertainties (e.g., the use of historical controls instead of contemporary data sets or the generation of patient-reported outcomes) with particular attention to those that will have the greatest impact on the decision. The debate as to whether HTA agencies could move from a traditionally “risk averse” position to a more “risk neutral” position (i.e., potentially more in line with the regulatory perspective) or whether HTA agencies should maintain their position and continue to require a detailed exploration, characterization, and reduction of uncertainties where possible is an ongoing dialogue (18). Acknowledging the two-way opportunity cost of mitigating uncertainty to avoid an “incorrect” funding recommendation versus enabling timely patient access to safe and effective treatments (i.e., incorrectly deciding to reject a technology that proves useful or adopting a technology that proves ineffective) is an ongoing issue for all stakeholders.

Transparent Early Dialogues

Related to but in many ways distinct from the concept of HTA life cycle management, early dialogues are a universally opportune time to start uncertainty management. Early dialogues with regulators and HTA bodies are well-established processes in which technology developers have the opportunity to discuss their research plans and gain advice and feedback on their planned approach. These activities can help identify uncertainties early in the process, along with the notion of how resolvable key uncertainties may be (19). This can influence the entire HTA process, from deciding if and when to conduct an HTA through to implementation of financial mitigation strategies such as MEA. Key considerations around early dialogues include challenges related to data sharing and confidentiality (with firewalls within agencies and the need for early dialogues to be nonbinding) and resource constraints (with the need for clear priority-setting criteria to choose the most important topics for early dialogue), and early patient and other stakeholder engagement is critical. The HTAi PCIG has created a toolkit for early patient input through the Patients Active in Research and Dialogues for an Improved Generation of Medicines (PARADIGM) project. This is a public–private, multistakeholder consortium that aims to provide a framework for structured, effective, meaningful, and ethical patient engagement along the life cycle of medicines. The framework includes tools (i.e., guidance, checklists, and fact sheets) for HTA agencies to adapt and use when engaging patients in early dialogue processes.

Even though early dialogue services are, by definition, confidential, broad efforts could be undertaken to improve transparency. For example, HTA agencies and industry stakeholders could document general learnings from early dialogue activities, with transparency to illuminate actions and decisions in each case (particularly regarding outcomes that matter most to patients). Similarly, there could be greater collaboration across countries to collect RWE for situations in which there are challenges in obtaining data, such as rare diseases.

Discussion and Next Steps

Given the virtual nature of the 2021 HTAi GPF as well as variability in the type and presence of uncertainty across settings and topics, it was not possible to conduct discussions that resulted in consensus statements or specific recommendations around considering and communicating uncertainty in HTA. Instead, the core themes that resonated with the membership are described above. In addition, a live poll was conducted on the final day that asked members for their “take home” messages related to the topic. From this, a “word cloud” was constructed (Figure 2). The most common messages focused on the notion that novel approaches to managing and communicating uncertainty must be clear and collaborative: “transparency,” “trust,” “participation,” and “multi-stakeholder engagement.”

Figure 2. GPF member “word cloud” on considering and communicating uncertainty in HTA.

There are immediate steps that can be taken to promote these core themes and take-home messages in uncertainty management and communication. For example, to improve the consistent consideration of uncertainties across jurisdictions, the notion of transparency should be applied to all stakeholders. The concept of “inclusion of the totality of evidence” should apply, where as much clinical evidence as possible is made publicly available, the definition of public data is purposely broad, and information described as “in-confidence” is limited to clearly commercially sensitive information so that the key uncertainties in an HTA can be openly discussed and clearly articulated by HTA agencies. Introducing programs that promote sharing of data (e.g., administrative data sets) across jurisdictions or limiting confidential data in submissions are examples of steps to promote transparency in these activities. The PCIG Summary of Information for Patients is also a mechanism to increase transparency in data.

Key challenges to reducing confidential information include competitive concerns in relation to alternative therapies nearing approval, which may be contrary to medical journal requirements, and the desire to produce HTA recommendations at or close to the time of product launch. However, if the full evidence base used to inform a decision cannot be presented transparently, then it follows that the uncertainties from that evidence base cannot be communicated transparently. Being able to clearly articulate where data are uncertain, including the magnitude of effect, how meaningful the effects are to patients, the true costs considered, and other factors are necessary for explaining the reasons for a particular decision. Managing this tension is likely to become more challenging as accelerated approvals increase, but developing a solution is critical to ensuring that HTA processes are viewed as credible and fair.

Another element closely related to transparency is that of health literacy for all stakeholders. In the practical short term, HTA agencies may consider standards of communication and ensure that all materials meet a basic and agreed level of health literacy (20). Sharing innovative methods across jurisdictions and promoting the use of visuals as a decision aid both for committees and for communicating uncertainty to the broader public should be feasible for most HTA agencies to achieve.

Transparency also relates to the need for HTA agencies to improve their reporting of how uncertainty is considered to make this more consistent and predictable across jurisdictions. The reporting could include features of the specific health system, culture, political structures, operational constraints, and other key contextual factors relevant to an HTA agency that may influence how uncertainty affects recommendation or decision making. Early dialogue (e.g., parallel consultations between the European Medicines Agency [EMA] and the European Network for HTA [EUnetHTA] (21)) was identified as a clear opportunity for considering uncertainty to ensure that the ultimate evidence package is designed to reduce as much uncertainty as possible. Both HTA agencies and manufacturers could increase documentation and learnings from their own early dialogue and scientific advice programs to share any practical limitations within their local context, and how these limitations influence the feasibility of addressing uncertainty as a result of early dialogue. More comparative work could be done across HTA organizations and industry to not only share data related to early dialogues where possible, but also to catalog and evaluate aspects of the early dialogue and scientific advice processes that may help increase efficiencies across jurisdictions. These learnings should be fed into HTA bodies that are considering developing their own early dialogue systems. This also applies to industry stakeholders, who should also be encouraged to present the sources of uncertainty and document the reasons for the decisions taken in developing their multistakeholder evidence packages and explain key aspects of trial design and outcome measurement to better clarify some of the uncertainties facing HTA agencies.

Patient and other stakeholder engagement (as opposed to consultation only) was also highlighted as critical for reducing uncertainty and helping to convey its presence in decision making. As mentioned, there are multiple tools now available to support participation (not just consultation), to encourage dialogue rather than one-way communication, and to characterize the impact of patient involvement (e.g., those developed by the HTAi PCIG), and these could be trialed by HTA agencies and stakeholder groups alike so that the quality and impact of engagement begins to improve and can be built into HTA processes more readily (https://htai.org/interest-groups/pcig/projects/current-projects/). In addition, transparency and clear communication regarding which data are needed for HTA but rarely systematically collected would be helpful. This would allow patient groups to ensure that the most relevant data on the most important patient impacts and outcomes are collected (e.g., through existing registries or development of disease-specific core impact sets) to address the needs of the HTA agencies in reducing uncertainty. Any implementation should be evaluated and learnings fed back to continually improve this important component of the HTA process.

All of these suggested next steps are clearly resource- and labor-intensive (though it may be argued that they could save downstream costs), and the capacity of HTA agencies to take on all of these additional activities is likely limited. As already alluded to, in the context of a full life cycle approach (i.e., from early dialogue through to post-decision-managed entry and beyond), criteria to prioritize the technologies that will be considered with a life cycle approach are needed. For example, technologies with the most potential for beneficial impact or those with resolvable rather than inherently unresolvable uncertainties could be prioritized, ensuring that choices about candidate technologies are systematic and transparent. Considering collaborative efforts (such as those that have been explored by the EMA and EUnetHTA) to initiate life cycle considerations would be an option to pilot key processes for this approach. However, the technology manufacturers also play an important role in these activities, and increased financial investment by industry in key aspects of life cycle management, such as early dialogue, iterative stakeholder conversations, and systems that support effective monitoring of MEAs, is needed to enable the move toward a life cycle approach. Greater involvement of funders (payers) and health professionals as evidenced in the “Idea, Development, Exploration, Assessment and Long-term follow-up” (IDEAL) framework and collaboration (22) (life cycle approaches to surgical interventions, invasive medical devices, and other complex therapeutic interventions) or government-led multistakeholder public–private partnerships such as Health Innovation Netherlands (https://www.healthinnovation.nl/about) can also help implement life cycle approaches.

Uncertainty in HTA is complex and multifactorial. Possible next steps to more openly and inclusively manage how uncertainty is considered in HTA and communicated to the wider public, as highlighted in this paper, require further vigorous debate and discussion, and additional topics and related recommendations may be identified. Those involved in various HTA societies and networks such as the HTAi, the ISPOR, INAHTA, EuNetHTA, HTAsiaLink, and the Health Technology Assessment Network of the Americas (RedETSA) should help drive the discussion forward. Development of an HTAi special interest group or joint society taskforce to evolve the topic further (including some of the methodological issues that were not the focus of the HTAi GPF) are possible mechanisms for maintaining momentum around this important discussion.