We welcome the interest of Grimm et al. (Reference Grimm, Pouwels and Ramaekers1) in the guidance of the HTAi-DIA Working Group that intends to support stakeholder deliberation on the systematic identification and mitigation of uncertainties in the regulatory-HTA interface (Reference Hogervorst, Vreman and Heikkinen2). In their letter, the authors put forth two arguments. First, they state that “it remains unclear if and how the state of the art on uncertainty in HTA was used to develop the guidance”, specifically that their paper “TRUST tool 2020 (Reference Grimm, Pouwels and Ramaekers3) was not cited”, and that it “bears non-negligible similarity” with the building blocks comprising decision-making uncertainty of the guidance. In essence, this comment relates to the process of developing the guidance. Second, they state that “it is unclear how the presented guidance improves upon this [the TRUST tool 2020].” As authors involved in the process of developing the HTAi-DIA guidance, we would like to reflect on these arguments.
Regarding the process of developing the guidance, we would like to reiterate that the guidance builds on the recommendations of the 2021 HTAi Global Policy Forum (Reference Trowman, Powers and Ollendorf4). It was developed by a cross-sectoral, interdisciplinary Working Group using a deliberate structured process. The steps and various methods deployed are documented in detail in the article (Reference Hogervorst, Vreman and Heikkinen2). We fully agree with the commentators that much has been written about uncertainty in and outside of the HTA domain. The literature that served as a basis for the discussions of the Working Group was, therefore, collected as part of the scoping process to determine the breadth of available information. It was – as clearly mentioned in the article (Reference Hogervorst, Vreman and Heikkinen2) – not a systematic, but a scoping review. The scoping review in combination with iterative, multi-stakeholder dialogues did, nevertheless, result in similar relevant topics that the commentators also mention in their response (i.e., identification, analysis and communication of uncertainty, and risk tolerance). Details on the scoping review and the included papers, among them the clearly cited TRUST tool 2020, are publicly available as part of the guidance (Reference Hogervorst, Vreman and Heikkinen2). Furthermore, 10 of the papers referenced by the commentators as missed are cited in our scoping review (Reference Grimm, Pouwels and Ramaekers3;Reference Grutters, Asselt, Chalkidou and Joore5–Reference Walker, Sculpher, Claxton and Palmer13), whereas four of the papers mentioned by the commentators were published after completion of the final Working Group meeting – as described in the guidance – in September 2022 (Reference Wolff, Qendri, Kunst, Alarid-Escudero and Coupé14–Reference Ayers, Cope and Towle17). The remaining papers mentioned by the commentators focus on health economic modelling, managed entry agreements, non-HTA or regulatory papers, which were not in scope for the purpose of the Working Group’s exercise.
Based on the majority of articles that were referenced by the commentators (i.e., focusing on health economic modeling, including the TRUST 2020 tool) and the topics that they considered to be missing (e.g., the scope and specificity of the methods for identifying and analyzing uncertainty), we believe that they – researchers from the Netherlands with a background in health economics – have a distinctly different perspective on approaching and addressing uncertainty in decision-making. The added value of the guidance is clear considering the emphasis on capturing different perspectives, and we consider the need to further stress the well-recognized importance of (early) dialogue between different stakeholders to ensure that there is mutual understanding and common language regarding uncertainty. Furthermore, the HTAi-DIA guidance acknowledges all relevant regulatory and HTA uncertainties which can be used in a proactive, rather than a reactive way. This means that besides improving mutual understanding, our guidance could be used to have discussions on the tools used to identify uncertainty, including, for example, the TRUST 2020 tool.
The guidance has been well-received by stakeholders across the HTA ecosystem, with a clear interest in using it in the regulatory-HTA interface. The HTAi-DIA Working Group is, therefore, confident to continue to address relevant topics in this field, such as applying the guidance in combinations of diagnostics and drugs and/or diagnostics and medical devices or medical devices only. Finally, as mentioned during all our communications, the HTAi-DIA Working Group always welcomes input from the broader HTA community and beyond, and we are open to those who wish to become active members of it.
Author contribution
Submitted on behalf of all the authors of the HTAi-DIA Working Group guidance on uncertainty management in regulatory and health technology assessment decision-making on drugs, who all signed off this letter: Milou Hogervorst, Rick Vreman, Inkatuuli Heikkinen, Indranil Bagchi, Inaki Gutierrez-Ibarluzea, Bettina Ryll, Hans-Georg Eichler, Elena Petelos, Sean Tunis, Claudine Sapede, Wim Goettsch, Rosanne Janssens, Isabelle Huys, Liese Barbier, Deirdre DeJean, Valentina Strammiello, Dimitra Lingri, Melinda Goodall, Magdalini Papadaki, Massoud Toussi, Despina Voulgaraki, Ania Mitan, and Wija Oortwijn.
Funding statement
For the writing of this letter, the authors did not receive any specific grant from any funding agency, commercial, or not-for-profit sectors.
Competing interest
I.H. is an employee of MSD and a stockholder of Merck & Co., Inc., Rahway, NJ, but acts as an individual contributor to this letter rather than an official company representative. The other authors declare no commercial or financial relationships that could be construed as a potential conflict of interest.
Open access
