Understanding innovation of health technology assessment methods: the IHTAM framework

Abstract

Adequate methods are urgently needed to guarantee the good practice of health technology assessment (HTA) for technologies with novel properties. The aim of the study was to construct a conceptual framework to help understand the innovation of HTA methods (IHTAM). The construction of the IHTAM framework was based on two scoping reviews, one on the current practice of innovating methods, that is existing HTA frameworks, and one on theoretical foundations for innovating methods outside the HTA discipline. Both aimed to identify and synthesize concepts of innovation (i.e., innovation processes and roles of stakeholders in innovation). Using these concepts, the framework was developed in iterative brainstorming sessions and subsequent discussions with representatives from various stakeholder groups. The framework was constructed based on twenty documents on innovating HTA frameworks and fourteen guidelines from three scientific disciplines. It includes a generic innovation process consisting of three phases (“Identification,” “Development,” and “Implementation”) and nine subphases. In the framework, three roles that HTA stakeholders can play in innovation (“Developers,” “Practitioners,” and “Beneficiaries”) are defined, and a process on how the stakeholders innovate HTA methods is included. The IHTAM framework visualizes systematically which elements and stakeholders are important to the development and implementation of novel HTA methods. The framework could be used by all stakeholders involved in HTA innovation to learn how to engage dynamically and collaborate effectively throughout the innovation process. HTA stakeholders in practice have welcomed the framework, though additional testing of its applicability and acceptance is essential.

Introduction

Health technology assessment (HTA) has become increasingly important throughout the world as a process to systematically evaluate properties and effects of a health technology with the purpose of supporting evidence-based decision making in reimbursement and clinical treatment (1). To guarantee good practices in HTA, adequate HTA methods are needed (2). HTA methods refer to all qualitative and quantitative methods relevant to the full scope of the HTA process (3), such as methods for evidence generation from clinical or real-world data (4;5), methods for synthesizing HTA evidence and modeling cost-effectiveness (6), and tools for dealing with uncertainty in multi-criteria decision making for health care (7). These methods vary by function and, if proven robust and implemented successfully, can improve the quality of HTA conducted throughout the HTA process.

The need for novel HTA methods becomes urgent when existing methods are not able to handle the complexity of emerging health technologies, which creates barriers for a systematic evaluation. Novelty here refers to the quality of being unusual in either structure or content of an HTA method, with the potential to resolve conflicts between traditional methods that HTA relies on and the quality of the HTA for emerging health technologies (8;9). For example, genetic testing, an emerging health technology to prognose individuals with high risks of genetic diseases, is ethically complex, so novel methods are needed to measure and value its ethical issues in HTA decision making (10). Digital health, another example of new technologies with unique features in data security and artificial intelligence, also needs specially designed methods to define and evaluate its HTA-related evidence (11).

To satisfy the urgent needs, HTA methods are developed and implemented, in other words, innovated, mainly in two ways: creation based on multiple disciplines of knowledge and improvement based on previously innovated methods. As the number of innovated methods increases dramatically, guidelines, such as the HTA core model (12), have been applied to inform HTA stakeholders (e.g., academics, healthcare professionals, HTA bodies, governments, patients, payers, and industry) on how to select HTA methods for different technologies in different settings. However, HTA stakeholders still lack an understanding of how to create or improve HTA methods. Consequently, stakeholders, especially those without an HTA knowledge background (such as patients and healthcare professionals), may lack consensus on which methods are urgently needed, how to innovate them, and, equally importantly, how they could engage in the innovation.

Therefore, the objective of this study was to develop a framework with two functions: to illustrate a generic innovation process that is applicable to all types of HTA methods and to illustrate how different HTA stakeholder groups can engage dynamically and collaborate effectively throughout the innovation process. We adopted a conceptual framework approach, which defines a network of concepts providing a comprehensive understanding of multidisciplinary phenomena and helping stakeholders understand knowledge from other disciplines (13). We considered this approach most useful to facilitate understanding of the complexities associated with innovating HTA methods.

Methods

The new framework was developed in two stages: first, identifying and synthesizing concepts of innovating HTA methods in two scoping reviews; and second, drafting the framework based on the concepts and refining the framework by gaining input from HTA stakeholders in the HTx project. This is an ongoing research project funded under the Horizon 2020 Framework Programme, with the aim to support patient-centered, societally oriented, real-time decision making for integrated health care throughout Europe (14). The flow diagram of developing the Innovation of HTA Methods (IHTAM) framework can be found in Figure 1.

Figure 1. Flow diagram of constructing the IHTAM framework. *Concepts of innovation indicate innovation processes and roles of stakeholders in innovation; ^‡A research project with an aim to develop and implement novel methods for patient-centered decision making using real-world data and machine learning techniques.

Identifying and Synthesizing Concepts of Innovation (Stage 1)

Our starting point was to identify concepts of innovation, defined as processes of innovation and stakeholders involved. Such concepts were considered likely to occur in two sources, therefore we performed two scoping reviews. The first source was literature on innovating HTA methods. Since we expected that lots of methods were innovated in the past through a variety of formats (e.g., frameworks, models, tools) and that the concepts extracted from different formats shared similarities, we limited ourselves to reviewing HTA frameworks. The second source was literature from scientific disciplines (defined as branches of knowledge) relevant to innovation, which might provide theoretical foundations for innovating HTA methods. For the two scoping reviews, we drafted protocols following PRISMA guidance (15) and conducted a pilot test to refine eligibility criteria, search strategies, and processes of data screening, abstraction, and synthesis.

Scoping Review on HTA Frameworks

HTA frameworks were identified in both scientific articles and gray literature. Documents were searched from PubMed, Embase, and Google Scholar. The search strategy included “framework” and “health technology assessment” (or “HTA”) in title and/or abstract. An article was included if it described a process in the methodology part on how an HTA method was developed, implemented, validated, or transferred; and excluded if it was not in English or full text was not available. The complete search strategy appears in Supplementary Table 1. According to the same in- and exclusion criteria, gray literature was searched from the Google Advanced Search and Web sites of seven international organizations which might report IHTAM, including the World Health Organization (WHO), the European Network for Health Technology Assessment (EUnetHTA), the Professional Society for Health Economics and Outcomes Research (ISPOR), the Society for Health Technology Assessment International (HTAi), the International Network of Agencies for Health Technology Assessment (INAHTA), the Institute for Clinical and Economic Review (ICER), and the National Institute for Health and Care Excellence (NICE). We searched for the “HTA framework” and took the first twenty items (sorted by relevance) of gray literature from each source because a pilot test showed that the first ten items were most likely to be eligible. Citations in eligible scientific articles and gray literature were also scanned for eligibility. Data screening was independently conducted by one author (L.J.) and cross-checked (10%) by another (M.A.H.).

Data items extracted from eligible studies included study characteristics (i.e., first author and publication year), a description of the innovation processes, and the stakeholders involved with their roles. Subsequently, data items regarding innovation processes or stakeholders involved were clustered and items with similar meanings were merged. For example, a process of “prototyping methods” and a process of “drafting solutions to a problem” were clustered as “design prototypes”; doctors and nurses were all clustered as healthcare professionals. Data items were extracted and clustered by one reviewer (LJ) and a random subset (ten percent) was checked by another (MH). Any discrepancies in data screening or extraction were resolved by discussion.

Scoping Review of Scientific Disciplines on Innovation

Concepts from scientific disciplines on innovation were identified only in scientific articles because a pilot search failed to identify eligible results in gray literature. An article was included if it provided a guideline on how to develop, implement, validate, or transfer an object; and excluded if the guideline was tailored to a specific object (e.g., school psychology), not in English, or full text was not available. The strategy of searching for concepts within scientific disciplines on innovation was also identical to that of the previous review, except for the search terms used (i.e., innovation, identification, research, development, implementation, validation, transfer, and generalization). Given the large number of items listed by databases, we only scanned the first 200 items (sorted by relevance) of each database as the pilot test showed data items after fifty of each database became less relevant. After identifying eligible articles, we further clustered them based on scientific disciplines. By scanning titles and abstracts of each article, we could identify theoretical foundations of the innovation processes and then determine which discipline an article belongs to. For example, a “framework for design thinking in health innovation” and a “design thinking framework for healthcare management and innovation” were clustered into a discipline called “design thinking.” The processes of data screening, abstraction, and clustering were also identical to those of in the first review.

Drafting and Refining the IHTAM Framework (Stage 2)

Brainstorming Sessions

Based on results of the reviews, the five authors organized six brainstorming sessions in three consecutive weeks to construct the framework. All opinions were recorded into notes by L.J. and reconfirmed by the authors who expressed them. Axial coding was used to identify how the concepts regarding innovation processes and those regarding stakeholder roles interact with each other, in other words, what roles HTA stakeholders could play and how their roles change along different phases of innovation. Selective coding was then used to select overarching concepts which all authors agreed to capture the essence of innovating HTA methods. Concepts without enough supporting data were deleted.

Stakeholder Input from the HTx Project

To further refine the draft framework, two further sessions, one face-to-face and one online, were organized on 7 February 2020 and 30 June 2020, respectively, during the HTx consortium meetings. All the participants of the consortium meeting received a notification of the rationale and schedule of the sessions 1 week before and were asked to confirm participation. The attendants were presented the latest version of the draft framework and asked to judge the relevance of the conceptualized innovation phases and stakeholder roles to the real-world practice of innovating HTA methods. Before the session, a questionnaire with open questions was sent to the attendants for the preparation and clarification of their opinions. L.J. recorded all the attendants’ opinions into notes and sent them e-mails for reconfirmation in case of any uncertainty. Open, axial, and selective coding was applied by L.J. to conceptualize the notes. To avoid the subjective coding bias, the coding process was reviewed by R.A.V.

Results

Identifying and Synthesizing Concepts of Innovating HTA Methods (Stage 1)

The flow diagram of identifying eligible studies and study characteristics of the two scoping reviews appears in Supplementary Figure 1 and Supplementary Table 2. Phases of innovation and stakeholders involved in innovation from the two scoping reviews are shown in Table 1.

Table 1. Phases of innovation and stakeholders involved in innovation from the two scoping reviews

^a Community indicates stakeholders who have problems and need innovative solutions.

^b Decision makers who decide on whether to adopt innovation.

^c Planners who consider contexts and stakeholders responsible for program adoption, implementation, and adoption.

^d Technical assistance experts who record implementation progress and advise on how to improve implementation processes.

^e Policy makers who develop policies regarding innovation.

Review on HTA Frameworks

Twenty eligible documents (see Supplementary Table 2) on innovating HTA frameworks were identified. The processes of innovation were clustered into nine phases (from “Identify needs for innovation” to “Transfer innovation”), and HTA stakeholders involved in innovation were clustered into seven categories: academics (mentioned most frequently, in 95% of the documents), healthcare professionals, HTA bodies, governments, patients, payers, and industry. In each phase of innovation, various categories of HTA stakeholders were involved, but we did not identify a pattern in the distribution of different HTA stakeholders across these phases. For example, of the five documents (25% of all identified) mentioning patient groups being involved in innovation, one disseminated a method (16); three tested HTA methods in case studies (17–19); and one evaluated method performance in practice (20).

Review on Scientific Disciplines on Innovation

Fourteen eligible documents from three scientific disciplines on innovation [design thinking (n = 4), implementation research (n = 9), and interdisciplinary research (n = 1)] were identified (see Supplementary Table 2). Innovation processes identified in this body of literature could be clustered into nine phases. Eight of the nine were similar to those from HTA frameworks, except for making decisions to adopt innovation (21–23), which was not mentioned by any HTA framework. In addition, compared to HTA frameworks, the three disciplines outside HTA provided more clarity on implications of each innovation phase. For example, innovation guidance from the discipline of design thinking implied that developers may observe other stakeholders’ behavior when identifying needs (24); guidance from implementation research implied that innovation should be disseminated clearly and concisely to stakeholders in various user-friendly formats (25). These detailed implications were not mentioned in HTA frameworks.

In the three disciplines, stakeholders were clustered into seven categories based on their roles in innovation. The most mentioned categories (developers, practitioners, and community) occurred across phases of innovation, whereas the less mentioned categories (decision makers, planners, technical assistance experts, and policy makers) occurred only in the last five phases (from “Disseminate innovation” to “Transfer innovation”).

Drafting the IHTAM Framework and Refining the Framework by Gaining Input from Stakeholders (HTx Project) (Stage 2)

Seven HTA stakeholders attended the face-to-face brainstorming session and six attended the online session. One stakeholder did not attend the sessions but completed the questionnaire for the online session. In all the fourteen stakeholders, academia accounted for eight, while representatives of HTA bodies, representatives of industry, and patients each accounted for two. The stakeholder characteristics are shown in Supplementary Table 3.

Phases of Innovation

All meeting participants considered the innovation phases from the two reviews relevant to innovating HTA methods, but some phases could be further split (e.g., “Identify needs for innovation”) or merged (e.g., “Disseminate innovation”), to be more understandable for them. They also advised to cluster the framework into three main phases, called “Identification,” “Development,” and “Implementation,” and to explain what tasks should be resolved through defining multiple subphases within each phase.

Roles of Stakeholders

The classic way of describing HTA stakeholders, for example HTA bodies, payers, patients, and industry, does not specify the roles they may take within innovation processes. In contrast, the categories of stakeholders that were derived from the scoping review of scientific disciplines are more widely applicable and better fit the purpose of roles of stakeholders within general guidance for innovation. One classic HTA stakeholder may take different roles within an innovation process. For example, academics could act as developers in one phase and as practitioners in another. Healthcare professionals could not only act as practitioners but also as decision makers.

To retrieve a small set of generic stakeholder roles for the innovation process, we further clustered the roles from the two reviews. Decision makers and technical assistance experts were considered being developers or practitioners; policy makers and community were not directly involved in developing or implementing innovation, but were affected by innovation, so we clustered them into “beneficiaries.” We thus defined beneficiaries, developers, and practitioners as the three generic roles that HTA stakeholders could play in innovating HTA methods, as shown in Table 2.

Table 2. Definitions of generic stakeholder roles in innovation

The HTx meeting participants agreed in principle that developers, practitioners, and beneficiaries could be tailored to contexts where HTA methods were innovated. But they emphasized that, in addition to the final framework illustrating an innovation process and stakeholders' roles, it needed to be made explicit how, in general, the classic categories of stakeholders, such as HTA bodies and patients, would translate to the stakeholder roles. After coding from the meeting participants’ opinions, we defined possible ways for HTA stakeholders to engage in innovation, as shown in Figure 2. HTA stakeholders can do so through two phases, which are called “role recognition” and “stakeholder discovery.”

Figure 2. Possible ways for HTA stakeholders to engage in innovating HTA methods. This figure illustrates possible ways for HTA stakeholders to engage in innovating HTA methods. The box on the left indicates HTA stakeholders (e.g., academics and HTA bodies) that can engage in innovation. The ellipsis at the bottom left indicates engagement of additional HTA stakeholder groups is also possible. The box on the right indicates the three roles HTA stakeholders can play in innovation (“beneficiaries,” “practitioners,” and “developers”). In the middle of the concept, map lists a two-phase process (“role recognition” and “stakeholder discovery”) on how HTA stakeholders play the three roles of innovation.

Role recognition indicates that HTA stakeholders first need to realize their roles in each phase of HTA method innovation. Stakeholder discovery indicates that, for each subphase, HTA stakeholders already involved in innovation may discover additional HTA stakeholders who are qualified as beneficiaries, practitioners, or developers. The stakeholders may, based on their own experience, evaluate who may be qualified for the three roles. After evaluation, those potentially qualified may be invited and contribute to the innovation. Since the tasks of beneficiaries, practitioners, and developers vary in different subphases of innovation, “role definition” and “stakeholder discovery” should be conducted iteratively throughout the innovation process.

The Final Framework

The final framework is shown in Figure 3 and illustrates a generic innovation process of HTA methods with three phases (i.e., “Identification,” “Development,” and “Implementation”). The three phases are distinguished by three colors, and each phase includes three subphases in white boxes. Underneath each subphase, the roles HTA stakeholders can play in that subphase are noted.

Figure 3. A generic process on how HTA methods are innovated. This concept map illustrates all key concepts of the IHTAM framework: (1) concepts relevant to a generic innovation process with three phases (i.e., “identification,” “development,” and “implementation,” which are distinguished by three colors) and nine subphases (in white boxes with numbers from 1 to 9); (2) roles of HTA stakeholders in innovation in each subphase (attached under each white box).

Phase 1: Identification

The identification phase, as the first phase of innovation, rationalizes the HTA method innovation and justifies stakeholders to be involved. In this phase, HTA stakeholders learn from past and present, imagine the future, and identify and evaluate the needs. “Learning from past and present” indicates that the stakeholders should acquire insight into limitations of current HTA processes. The commonly used techniques include surveys, interviews, literature reviews, or observations on how an HTA progress is conducted (11,30–32). A recommendation on how to identify up-to-date limitations is to gain feedback from practitioners who used traditional methods and beneficiaries who are affected by them. As emphasized by the design thinking theory, stakeholders may not really realize a limitation themselves (33;34). Still, limitations may be identified after observing and analyzing how practitioners act in practice (35). “Imagine Future” refers to picturing what future HTA processes looks like, and identifying enablers and barriers for the imagined future. One way to achieve this is to construct future scenarios through round-by-round brainstorming with the techniques such as group interviews and surveys (11;31;36). Future imagining could be conducted together with learning from past and present. The identification and evaluation of needs, as the third subphase, is the goal of the identification phase and the premise of developing HTA methods. Based on a gap identified by comparing future HTA scenarios with current HTA practices, HTA stakeholders may evaluate the heterogeneity of contexts where gaps are identified. The various contexts in which HTA is conducted, such as different types of health technologies, disease areas, or geographic areas, need to be considered as corresponding needs may vary. Once needs are identified, stakeholders may decide whether existing methods can be improved or novel methods need to be developed. A decision could be made by investigating transferability opportunities, as suitable methods may already exist in other contexts. The methods innovated originally in other disciplines of knowledge may be worth studying if they have the potential to be applied in HTA. A challenging task throughout the identification phase is the participation from a large group of stakeholders with different roles. Not only academics but also any potential stakeholders qualified as potential practitioners and beneficiaries could identify or evaluate the needs. In practice, stakeholders except academics are less involved in needs identification or evaluation (see Table 1). Our suggestion is adopting a regular procedure of “stakeholder discovery” and “role definition,” as illustrated in Figure 3. In this way, initially involved stakeholders, for example academics, could identify and invite other stakeholders with clarified distinguished roles.

Phase 2: Development

To develop an HTA method robustly, several concerns should be considered. First, resources for innovation should be managed in a good way. This usually begins with human resource management, that is, defining a group of method developers from a range of HTA stakeholders. Developers may set the priority for the needs that a novel method addresses, then establish an external research communication mechanism and avoid duplication of efforts of development. Academics could lead the group of developers, but other HTA stakeholders could also take up the role, depending on the contexts (18;37). Then developers should make agreements on the concentration and allocation of all the other resources, such as time, finance, and knowledge (23;25;38). A typical way of resource management is to conduct a feasibility analysis to evaluate what resources are needed and whether resources are available (39). Second, if method development is feasible, developers may design a method prototype and its derivative versions based on the heterogeneity of needs to improve the method capability that can be transferred to various HTA contexts. Feedback from practitioners and beneficiaries should also be reflected in method development, as innovation successes largely depend on how easily a method can be implemented (33). Therefore, developers need systematic approaches to gaining feedback regularly from beneficiaries and practitioners. One solution could be “ideation,” a commonly used process in the design thinking theory, which synthesizes insights from multiple stakeholders for addressing design challenges (40;41). The final subphase of development “Pilot testing” is to validate HTA method prototypes. Before applying the prototypes to practice, developers may first disseminate method prototypes to practitioners and engage those who feel interested in the methods being developed. These practitioners then implement methods in pilot contexts (16;36;37). One concern is how to identify and organize pilot case studies that could simulate real-world practice while avoiding consequences in case of any error caused by design flaws, lack of transferability, or wrong operations. Method validity could be judged by all stakeholders in a structural way (27;29;32;37).

Phase 3: Implementation

A method innovation process is not complete until a method is implemented successfully. During the implementation, as what implementation science often stresses, stakeholders need to plan for implementation, apply a method to practice, then transfer it to other contexts after validation (34). Any developer or practitioner involved in method development may contribute to diffusion (e.g., scientific publications and conferences) or dissemination (e.g., training) of methods to practitioners in real-world practice (34). Implementation strategies may also be developed, in which all resources needed for conducting and monitoring implementation are considered (24;38;41). Strategies need to be tailored for different contexts where HTA is conducted. One challenge of planning for implementation is how to motivate real-world practitioners and beneficiaries to adopt the novel method in practice, as any reluctance to method uncertainty or misunderstanding could deter the adoption.

Once a method is adopted, concerted effort is required by all stakeholders who are qualified as practitioners to implementing the method (37). Developers, with knowledge of a novel method, should continuously provide technical assistance and work with practitioners to adjust implement strategies to various contexts when necessary (22;42;43). A feedback loop, which cycles through the method application by monitoring, adoption, and tailoring, could make an HTA method more sustainably entrenched within a context (37). Regular debriefing of implementation progresses could be performed for the later validation purpose (23).

Finally, in the last subphase “Test & Transfer,” the performance of a method should be tested with an intention of further innovation. Developers need sound approaches to systematically test the validity of HTA methods, then report the results transparently to all stakeholders. The results worth reporting include outcomes of an HTA method, the extent to which a method is adopted by practitioners and beneficiaries, and the quality of implementation strategies (21–23). Practitioners from other contexts may be invited, as they could help judge method transferability and point out potential concerns during the transfer. Group decision making is required on whether the method is robust, and in what condition it can be transferred (22). Finally, a discussion may be initiated to justify the necessity for another round of innovation.

Discussion

We developed a conceptual framework that provides an understanding of how to innovate HTA methods. The IHTAM framework illustrates a generic innovation process on how to identify needs for, develop, and implement HTA methods. The framework also outlines a process on how HTA stakeholders can engage in innovating HTA methods.

Our framework adds value to HTA's good practice for several reasons. First, the framework contributes to a collaboration of HTA stakeholders from various disciplines. By defining three generic roles (beneficiaries, practitioners, and developers) of innovation and tasks of each role in each phase of innovation, the framework prompts HTA stakeholders to think beyond the traditional view on stakeholder roles whether, at which phase(s), and for which role(s) they are qualified for innovation.

Second, as the first to provide a general understanding of innovating HTA methods, the framework serves a foundation for constructing or improving more specific guidance on innovation. Some specific guidance does already exist. For example, a guideline was developed for developing, implementing, evaluating, and reporting discrete event simulation, a novel computer-based modeling that is increasingly applied in the HTA context (44). The guideline described relevant concerns and best practice recommendations throughout the innovation process. Another example is a report developed by the ISPOR to guide the developing and implementing of a type of HTA decision-making methods—multiple-criteria decision analysis (MCDA)—to support healthcare decisions (45). The report outlines an eight-phase process of MCDA development and implementation. Although these guides focus on the innovation of one type of HTA methods, our framework provides a general understanding of innovating all types of methods.

Third, the framework promotes consideration of key challenges that may exist in innovating HTA methods. It lists phases of innovation that may be implicitly known but not explicitly considered currently by HTA stakeholders. For example, the subphase “Apply a Method to Practice” implies that practitioners may decide on whether to apply a method to practice. Apart from considering who are qualified as practitioners, HTA stakeholders in a specific context may consider what criteria should be used for decision making. Attaching importance to challenges of innovation contributes to method validity and implementation success.

How to Use the IHTAM Framework

The IHTAM framework has the potential to become a starting point for HTA stakeholders to understand their roles in innovating HTA methods and we consider all HTA stakeholders as a potential audience of the framework. It is important to realize that (sub)phases of the IHTAM framework do not necessarily occur sequentially, and a specific innovation process should always be defined for each method innovated. To determine the most appropriate process and roles, stakeholders always need to consider actual conditions and initiate detailed discussions. The function of the IHTAM framework in determining an appropriate innovation process or roles is to explicitly illustrate what aspects of innovation need to be considered. In summary, we recommend considering the following when using the IHTAM framework:

1. (1) Consider all (sub)phases and three innovation roles within the IHTAM framework and judge their relevance to the methods to be innovated.

2. (2) Discuss whether additional (sub)phases or roles of innovation apply.

3. (3) Construct a tailored innovation framework and consider challenges of innovation to be addressed.

4. (4) Evaluate the qualification of HTA stakeholders for innovation and facilitate collaboration.

Limitations

One limitation of our study is that the article selection of gray literature may be difficult to replicate. In our search strategy, only the first twenty items of “HTA frameworks” listed on the Google Advanced Search, and the seven international organizations were included. The sequences of items from the above-mentioned sources can be influenced by their searching algorithms. Particularly, the Google Advanced Search is highly influenced by a user's own preference. However, this limitation would not cause much impact on the coding, as only one of the 34 eligible literature documents was sourced from the gray literature (Supplementary Table 2). For data extraction of the scoping reviews, one reviewer independently scanned all titles and abstracts, whereas the second reviewer checked only 10% of them. This might cause exclusion of some eligible literature but might not influence the results of conceptualization. The reason is that we considered the included literature already sufficient (n = 34) for the coding purpose, as we kept adding literature until saturation was reached, in other words, until the point where new literature did not provide any additional information. Furthermore, there are several limitations on the framework applicability. For the review of HTA methods, only the “framework” type of methods was included, so the applicability of the IHTAM framework might be limited when applying to other types of methods, such as HTA models. For the brainstorming sessions, we did not invite HTA stakeholders outside the HTx project. Even within the project, we relied on a relatively low number of HTA stakeholders to confirm the usefulness of the framework. Another limitation is that not all HTA stakeholder groups, such as payers, were invited for input. Hence, uncertainty still exists on whether the framework is accepted by HTA stakeholders in various contexts. Still, recommendations provided by the IHTAM framework are worth considering, because it can serve as a starting point to illustrate the complex innovation process and how it is related to HTA stakeholders. Although the IHTAM framework will not function as a quality checklist that can be rigidly followed, the way we conceptualize the method innovation and the relevant challenges we propose is worth noting for all types of HTA methods and for all HTA stakeholders.

Based on the IHTAM framework, it is possible to build an in-depth pathway to further identify and solve particular challenges in innovation. We thus recommend future efforts to test the applicability and acceptance of the IHTAM framework in case studies of innovating HTA methods in various contexts.

Conclusions

The IHTAM framework provides an understanding of how to innovate HTA methods and it helps HTA stakeholders better understand how to engage in innovation by knowing what different roles they can play in complex contexts of innovation. We believe the framework may add value to the development of robust HTA methods and effective implementation, which helps meet the needs for novel HTA methods due to emerging health technologies.