Considerations for transferability of health technology assessments: a scoping review of tools, methods, and practices

Abstract

Health technology assessment (HTA) is commonly used to guide evidence-informed decisions to optimize resource use, prioritize policies, and support countries to achieve universal health coverage. Producing HTAs requires time, scientific expertise, and political commitment, but these are not available in all settings – especially in low- and middle-income countries (LMIC) where HTA processes may be less institutionalized. Transferring and adapting existing HTAs to local settings may offer a solution while reducing duplication efforts. This scoping review aims to provide an overview of tools, methods, approaches, and considerations which can aid HTA transfers. We systematically searched (from 2005 to 2020) six databases and, using predefined inclusion criteria, included twenty-two studies. Data extraction followed a structured process, while synthesis was more iterative. We identified a common approach for HTA transfers. It follows the de novo process of undertaking original HTAs, but with additional steps to assess relevance (applicability), quality, and transferability, as well as steps to adapt parameters where necessary. The EUnetHTA Adaptation Toolkit was the only tool that provided guidance for adapting multiple HTA domains. Other tools were specific to systematic reviews (n = 1) or economic evaluations (n = 12), where one provided guidance for systematic reviews of economic evaluations. Eight papers reported transferring an HTA, with only one transferring to an LMIC. Finally, we reported issues that may facilitate or hinder transferability. In conclusion, we identified fourteen transfer approaches in the form of guidance or checklists, but harmonized and pragmatic guidance for HTA transfers to suit settings with limited HTA capacity seems warranted.

Introduction

Health technology assessment (HTA) can facilitate transparent, accountable, and evidence-based decisions that support equitable and efficient allocation of healthcare resources (1). However, producing HTAs requires considerable time and resources, and reusing existing HTAs can be helpful, particularly for urgent policy decisions (2). The process of transferring existing HTAs between different settings (i.e., countries, regions, or HTA agencies) could then accelerate the production of HTAs, as well as; reduce duplication, improve knowledge sharing, and aid countries with fewer resources for HTA (3–5).

Generally, HTA production follows four steps, namely “Topic Identification, Selection and Prioritization (TISP),” “HTA analysis,” “Appraisal,” and “Implementation” (6). There is consensus that stakeholders (i.e., patients, providers, policymakers, etc.) should be involved in each step of the HTA process, as this brings many benefits, such as the inclusion of relevant outcomes regarding the quality of life and cost-effectiveness, understanding uncertainty of data and results, and further implementation (7;8). This is especially true when transferring HTAs where most or all evidence is not from local sources and the involvement of stakeholders will ensure local relevance. As such, HTAs may inform a broad range of decision-making processes, including reimbursement, coverage, procurement, clinical guidelines, priority setting, and public health programs.

The end-products of HTA processes often vary but may be divided into four main types: HTA reports or full HTAs; mini-HTAs; rapid assessments; and national appraisals. Each can support different decision-making processes depending on the context; local requirement; technology under assessment; and the type of decisions (9). HTA reports are the most comprehensive, describing the health technology and its current use, evaluating its clinical effectiveness and safety, and often including an economic evaluation. HTA reports may include additional parts, or so-called domains, that assess Ethical, Social, Legal, or Organizational aspects (ESLO) (10). The variability of HTA content and format might benefit local decision-making but limits the ease of HTA transfers.

When considering HTA transfer, it is important to determine whether the underlying evidence is relevant (applicable), generalizable, or transferable. There is currently no consensus on how to define these terms (Table 1) (11). Berg et al. (12) proposed two steps to guide the transferability of HTAs. First, an applicability, or relevance, assessment investigates if “the health technologies considered in the HTA can be replicated and evaluated in the new context?.” This is followed by a generalizability assessment, a mostly scientific effort, that considers whether “the effect size found in the HTA will be retained when implemented in the new context?” (12). If an HTA is applicable and aligns in its current form, it is considered “generalizable.” However, more often interventions included in HTAs are applicable and transferable to the setting, but the evidence is not generalizable. Some HTA domains are inherently more generalizable or transferable than others. Findings from clinical trials used to assess efficacy and safety are often generalizable if target populations and conditions are similar, whereas costs and resource use in economic evaluations, as well as the ESLO domains, are often not generalizable nor easily transferable, adding to the complexity of transferring HTAs (13).

Table 1. Definitions of transferability and generalizability

Note: The EUnetHTA Glossary provides various definitions for transferability and generalizability, because there is no consensus among the various country partners – here, we present the definition defined in the EUnetHTA Adaptation Toolkit.

Despite issues related to transferring evidence to a new context, most HTA agencies already reuse existing evidence to respond to local needs. The European Network for HTA (EUnetHTA) has explored ways of standardizing methods and reporting and HTA transferability, resulting in products like the HTA Core Model (10), a methodological framework to collaboratively produce and share HTAs, which can indicate the level of transferability for each assessment element (10), and the EUnetHTA Adaptation Toolkit, which can aid HTA agencies to transfer HTAs produced elsewhere (14). The objective of this review is to provide an overview of HTA methods, approaches, or tools like the EUnetHTA toolkit to transfer HTAs. We also investigated other issues that might aid or hinder HTA transfers specifically in low- and middle-income countries (LMIC).

Methods

We followed guidelines for scoping reviews by Arksey and O’Malley (15) and Peters et al. (16). We aimed to answer three research questions: (i) What methods, including approaches, checklists, or tools, are available for the adaptation and adoption of HTAs?; (ii) What factors of these methods aid or hinder the process of integrating an existing HTA in a new context?; (iii) Are there any methodological gaps associated with these methods, specifically for LMICs? The protocol is available on the Norwegian Institute of Public Health (NIPH) website (17). We opted for an iterative process to allow for flexibility when reporting our findings. We made minor changes to the data charting (Supplementary Material S1) and carried out data synthesis and reporting related to the main themes identified in the included articles.

Eligibility criteria

We were primarily interested in three types of studies, namely studies that transferred an HTA from one setting to another, along with a description of experiences arising; studies that described a tool, approach, or method for transferring an existing HTA, including an explanation of how the tool could be used in practice; and systematic reviews of various tools for HTA transfers. There were no restrictions applied to the type of HTA report, and the HTAs could include multiple domains or only a specific domain such as clinical effectiveness, safety, or economic evaluation. We excluded studies providing methods that mainly reused HTA recommendations and not the underlying assessments. For instance, Multiple-Criteria Decision Analysis (MCDA) or deliberative processes for priority setting are tools primarily developed to be used by decision-makers (HTA users) and were considered outside of the “HTA-analysis” step, as well as mini-HTAs, hospital HTAs, and HTA fast-tracking. Lastly, studies needed to be accessible in full text, related to public health, and published in English after 2005 to be eligible for inclusion.

Search strategy

To identify relevant literature, an information specialist helped to develop the search strategy and performed literature searches in November 2020 in six electronic databases: Ovid Medline, Embase, Cochrane Database of Systematic Reviews, Scopus, Epistemonikos, and the Cochrane Methods Methodology Register (Supplementary Material S2). We did not search in reference lists nor for grey literature but included conference abstracts from the past three years. Additionally, we sent a short questionnaire to members of the International Network of Agencies for Health Technology Assessment (INAHTA) about whether they knew or used any transfer tools or had any transfer experiences, in order to identify tools in development or not publicly available (Supplementary Material S3).

Study selection, data extraction, and charting

Articles fulfilling the inclusion criteria were uploaded to Covidence for title and abstract screening performed by four reviewers (LF, EP, IS, and LC), where at least two reviewers independently screened references. EP and LF read and screened the full texts, while IS and LC verified their decisions. Disagreements were resolved through discussion or by consulting a third independent reviewer. EP and LF used Microsoft Excel (2016) to document, extract, and chart data. Items on the standardized data extraction sheet were guided by the research questions (Supplementary Material S1). We extracted or summarized data on all relevant items from each article. Afterwards, articles were grouped per HTA domain(s) and the data were narratively summarized. The strengths and weaknesses were identified, thematically grouped, and described, with specific attention to LMICs.

Results

The search strategy identified 1,323 unique references. Following screening, we read seventy-two full texts, of which nineteen articles were eligible for inclusion (Figure 1). We included three additional articles, because the search strategy identified only one of three articles from an article series (18–20), and one article on the TRANSFER approach was published after the search (21). The TRANSFER approach was developed following a scoping review (22) identified from the search strategy. Excluded studies and the reasons for exclusion are listed in Supplementary Material S4.

Figure 1. PRISMA flowchart of study selection process. The number of records identified per database are Ovid Medline (n = 464), Embase (n = 519), Cochrane Database of Systematic Reviews (n = 58), Scopus (n = 685), Epistemonikos (n = 304), and Cochrane Methods Methodology Register (n = 101).

.

The eligible twenty-two articles referred to HTA transferability in their approach, method, or proposed tool (Supplementary Material S5). Eight articles described their experience of conducting an HTA transfer using a tool (23–30), twelve articles described transferability tools (4;18–21;31–37), and two articles provided an overview of tools aiding transferability (22;38). No additional tools were identified by the INAHTA questionnaire (seven respondents) or among the conference abstracts (Supplementary Material S3).

The included articles were heterogeneous in their scope, outcomes, and approaches to HTA transferability. Firstly, we focused on the structure of the HTA transfer process (Figure 2). Then, we summarized three transferability approaches that guided the whole HTA process, namely: the EUnetHTA Adaptation Toolkit (14) along with the derived Model for Assessment of Telemedicine (MAST) (34); the TRANSFER approach for systematic reviews (21); and systematic reviews of economic evaluations (18–20). Next, we listed relevant “catalyst” tools that may aid transferability assessments in economic evaluations (Table 2). We then discussed key issues that affect HTA transfers.

Figure 2. “Common» structure of HTA transfers.

Table 2. Tools to assess or guide transferability of economic evaluations

Note: We excluded the SBU checklists (47) and checklists not being specific to transferability (e.g., the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist (49) or model quality checklists as Caro et al. (42) and Philips et al. (41)).

The “common” structure of HTA transfers

There is no consensus on how to structure HTA transfers; however, our findings indicated overlap and common features between HTA transfers and processes for de novo HTA production. Figure 2 illustrates the overall structure of an HTA transfer. To summarize, after having decided to opt for an HTA transfer (step 1), a relevant HTA or research underlying a specific HTA domain needs to be identified. This process commonly uses pre-defined selection criteria and follows procedural recommendations as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement (39). The checklist by Grutters et al. (40) can help to determine and frame the relevance, or applicability, of an HTA to the identified research question (step 2). The identified HTA(s) is then assessed for methodological quality, risk of bias, and transferability (step 3). Step 3 is crucial in the transfer process, as transferring a low-quality product might lead to ambiguous, misleading, or even wrong conclusions. At this step, approaches start to diverge, for instance in the order of the quality, risk of bias, or transferability assessments. Some approaches even fuse steps 2 and 3, combining applicability and partial quality assessments into one. What most tools have in common, however, is that they prepare for transfers like a catalyst in a chemical reaction. An additional step might be added to identify transferability factors, which refers to factors that need to be adapted or replaced with local data. After identifying transferability factors, and when necessary, adapting or replacing them (sub-step 3), relevant data can be extracted (step 4). Then, the results are reported (step 5), discussed, and interpreted (step 6). Finally, the appraisal (step 7) and implementation in the local setting (step 8) conclude the HTA transfer process.

EUnetHTA adaptation toolkit

The EUnetHTA Adaptation Toolkit consists of two sections (4;14;31–33). First, eight speedy sifting questions (on topic, language, description of the technology, methods, scope, peer review, conflict of interest, and timeliness) appraise the applicability of the identified HTA. After deciding to continue the adaption process, the second section includes questions on relevance, reliability, and transferability for five “transferable” HTA domains (use of technology, safety, effectiveness, economic evaluation, and organization) (14). The MAST Application is derived from the Adaptation Toolkit, but with guidance questions specific to HTAs on telemedicine applications (34).

Macpherson and Thompson (24) describe their experiences using the toolkit to adapt two relative effectiveness assessments produced by EUnetHTA to inform decision-making in the National Health Service in Scotland. The modular structure of the toolkit makes it possible to use the whole or only parts thereof (14); and Gorry, McCullagh, and Barry (23) used questions from the economic evaluation domain to assess the generic and specific transferability of published economic evaluations on treatments for advanced melanoma to the Irish setting.

TRANSFER approach guides interpretation of transferability in systematic reviews

The TRANSFER approach mirrors the standard systematic review process in seven stages and provides guidance to assess and integrate the transferability of review findings. For instance, templates for collaborating with decision-makers and identifying context-specific “transfer” factors are included (21).

Systematic reviews of economic evaluations

A systematic review of economic evaluations collects economic evidence about specific health interventions to inform evidence-based decisions. The article series by van Mastrigt et al. (18–20) describes the process, with expert best-practice recommendations for each step, as follows: (i) initiate the review; (ii) identify and select economic evaluations; (iii) data extraction and assessment of the risk of bias and transferability; (iv) reporting results; (v) discussion and interpretation of results. Economic evaluations must “pass” the quality and risk of bias assessment before an assessment of transferability is relevant. Wijnen et al. (20) provide a comprehensive overview of checklists that can facilitate these quality assessments. Notably, they recommend the Philips Checklist (41) for assessing the risk of bias in model-based economic evaluation. However, the Philips Checklist is rather long, so, when over ten economic evaluations are under review, the checklist by Caro et al. (42) was recommended. For trial-based economic evaluations, Wijnen et al. (20) recommended the BMJ (43) or CHEC-extended (44) checklist. Further, identified economic evaluations may be compared to local guidelines to fit methodological standards and ensure relevancy. The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) has collected thirty-three country-specific guidelines that may be used for this purpose (45) [access at: https://tools.ispor.org/peguidelines/]. Lastly, Wijnen et al. (20) highlighted the Welte checklist (46) as a convenient transferability checklist.

The search strategy identified three papers conducting systematic searches to find and transfer economic evaluations. Checklists were primarily used to assess if results could be adopted (referring to a direct transfer without any changes) in the local setting. Only the article by Nystrand et al. (25) used the “Van Mastrigt” guidance (18–20). They used multiple tools for the transferability assessments, including a country-specific checklist from the Swedish HTA Agency (SBU) that covered transferability items other checklists did not consider (e.g., the feasibility of implementation, standard practice in the comparator, and health financing mechanism) (47). Van Haalen, Severens, Tran-Duy, and Boonen used a combination of checklists and approaches like the disease-specific reference case, the Welte checklist (46), the Drummond-ISPOR approach (48), and the Phillips checklist (41). Ruggeri et al. (27) used pre-defined criteria and the Augustovski checklist (5) to assess generalizability. All studies found that economic evaluations could be transferred although adaptations were required, for instance replacing unit costs.

Catalyst transfer tools for economic evaluations

Another way to transfer economic evaluations is exemplified by the more ad-hoc approaches that identified or had access to existing economic evaluations and adapted underlying models to fit the new setting (28–30). These approaches followed the steps from Figure 2, using a variety of tools to assess applicability, quality, bias, or transferability. Ademi et al. (28) used multiple tools and adapted the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist (49) to predefine exclusion criteria for the selection process (step 2), while Essers et al. (29) used the knock-out criteria in the Welte checklist (46) as a quick-sifting tool. If economic evaluations failed to report items from the CHEERS (49) or Welte checklist (46), they were deemed non-transferable and excluded. After relevant economic evaluations were identified and selected, Ademi et al. (28) used quality assessments to judge the methods’ characteristics and other aspects, while neither Alshreef et al. (30) nor Essers et al. (29) included quality assessments. Both used models previously applied in another context, and this validation was considered appropriate. However, Essers et al. (29) did use the Urdahl (50) and EURONHEED Checklist (35), and Alshreef et al. (30) used the Mullins checklist (51) to identify factors that could limit the transfer (step 3). Identified factors included costing parameters, resource use, discount rates, and outcome measures (i.e., mortality, utility, and health-related quality of life). All papers adjusted factors that could limit transferability by replacing them with local data if available. Otherwise, expert assumptions or international studies were used. Table 2 provides an overview the identified checklist that can aid the transfers of economic evaluations.

HTA transferability issues to consider

The articles included in this mapping all reported on factors and issues that may affect the transferability of HTAs. For instance, it is essential to identify and access relevant HTAs, research papers, or models to avoid duplication (20;23;25;28;29;38). Differences in the context of the technology under review (i.e., resources available, the standard of care, type of health system, and population) further hinder the transferability of HTAs (30;34;37). Replacing key parameters with local data can help to ensure relevant estimates, but the country-specific data are often lacking in resource-limited settings. Sensitivity analyses and other estimation techniques could help fill any data gaps, but will require knowledge and skills (34;35;38). Additionally, knowledge of HTA, and specifically appraisal skills, is essential. HTAs that use poor-quality data will only have limited use for a transfer, as this might perpetuate poor HTA recommendations (4;31;32). Lastly, differences in HTA practices also affect the ease of transferability. Hence, the challenges of HTA transfers should not be underestimated as they still require time, scientific expertise, and other resources.

Table 3 presents an overview of all the identified transferability issues, which we summarized in five overarching themes: (i) lack of access to original studies or models, (ii); poor quality of HTAs, underlying studies, and models; (iii) lack of local data limits possibility to adapt existing HTAs; (iv) local HTA requirements ensure relevancy, but diversify end-products; and (v) significant resources are required to conduct HTA transfers.

Table 3. Factors that may influence the transferability of HTAs

HTA, health technology assessment; PRISMA, preferred reporting items for systematic reviews and meta-analyses.

Discussion

This scoping review identified the six steps commonly followed when transferring HTAs to a new context (Figure 2). After initiating (step 1), searching, and selecting (step 2) an applicable HTA, one or more tools are used to assess the transferability (step 3). These tools, sometimes adapted from their original purpose (e.g., CHEERS being commonly used to ensure comprehensive reporting (49)), are then used to assess the quality and transferability of the HTA. Methods are available to identify, adapt, or replace transferability factors when evidence is not generalizable, or transferability can be improved (step 3b). The guidance for the steps following these assessments, for instance, on how to interpret added uncertainties of transferability, is limited (steps 4–6). Nor is there a clear consensus on when to use which tools to assess transferability (25;38;52). Most of the identified tools were checklists relevant for economic evaluations, while only a limited number of tools were suitable for systematic reviews that assessed the clinical effectiveness and safety of health interventions. The EUnetHTA Adaptation Toolkit and the derived MAST application were the only tools including checklists for multiple HTA domains. However, no guidance was provided for the ESLO domains, as these were deemed less appropriate for transferability given that they are highly contextual and require local information (4;31–33).

We identified various issues hindering the transferability of HTAs. Most importantly, HTAs inform decisions locally and, for diverse reasons, existing HTAs may not fit with the settings’ needs, policy questions, or methodological standards. Many of the approaches identified in this review tried to overcome these issues. For instance, Grutters et al. (40) provided a checklist that can be used to assess the applicability of HTAs to the decision problem. Secondly, the TRANSFER approach highlighted the benefit of engaging stakeholders early as they can play an important role in refining research questions and identifying those factors that affect transferability. Additionally, the TRANSFER approach allows to run a sub-analysis specific to these transferability factors to obtain estimates relevant to the local setting. (21). Thirdly, country guidelines or reference cases (e.g., SBU checklist (47)) could help to match existing HTAs to local requirements and needs. Likewise, disease-specific reference cases may aid transfers (25). Fourthly, local data can replace parameters in transferred models or HTAs. For instance, ISPOR recommends minimally replacing price in economic evaluations (48). Both quantitative and qualitative methods have tried to identify those transferability factors which significantly affect outcomes (53;54). Goeree et al. (54) identified over seventy-seven transferability factors showing that adapting parameters in economic evaluations is still a complex task. Lastly, even with existing databases, such as the INAHTA HTA database, Cochrane library, TUFTS database for economic evaluations, and many more, access to relevant evidence is still an issue in HTA transfers.

We identified no transfers between LMICs and only one paper described a transfer to an LMIC (30). However, limited capacities and potentially high up-front costs for conducting and implementing HTAs might be disincentives to institutionalizing and producing HTAs and HTA transfers (55). None of the tools identified in this review provided pragmatic guidance on how to overcome these challenges. Collaborations illustrated by Alshreef et al. (30) might help to bridge the gap and strengthen capacities in countries with limited resources for HTAs.

Implications for practice

Our findings indicated that the EUnetHTA Adaptation Toolkit was the most comprehensive and versatile, including separate checklists for five domains (use of technology, safety, effectiveness, economic evaluation, and organization). However, other tools might fit better depending on the type of evidence to be transferred. For instance, Alshreef et al. (30) omitted the EUnetHTA Adaptation Toolkit (14) because it was not specific to model adaptation. The Mullins checklist (51) was selected given it fitted best with their criteria, which included relevance to model adaptation, endorsement from a respected organization, compatibility with the International Decision Support Initiative’s reference case for economic evaluations, transparency, inclusiveness, length, and external validity. Hence, which tool to select to transfer HTAs in any setting is not straightforward and careful consideration is required.

All HTA transfers will face trade-offs between minimizing bias and maximizing precision when selecting evidence. These issues are often magnified in LMICs, which have limited access to appropriate local data, and where research from HICs is usually not as easily applicable or generalizable. Therefore, low resource settings must be pragmatic, as HTA transfers will never be perfect. Local or generalizable studies from other settings might not exist, whereas transferring and adapting applicable studies that answer the right question, even if the study population is small or quality is low, can still provide valuable insights (56). There is a need to strengthen capacity and provide pragmatic guidance which discusses the trade-offs, benefits, and disadvantages of HTA transfers. Additionally, involving stakeholders is essential as they can provide expert information, ensure relevant research questions are asked, inform about transferability, and comment on the uncertainty of transferred HTAs (7;8;21).

HTA transfers are only one of the approaches which could benefit LMICs. There are various other pragmatic and rapid HTA approaches, also referred to as Adaptive HTA (aHTA), that reuse existing evidence to inform decisions, such as expedited processes, adaptation of global datasets, reuse of reviews, and price-benchmarking (2). These aHTA approaches require a diverse range of knowledge and skills to interpret the findings and understand uncertainty potentially added by the adaptation, or transfer, processes. There is a need for more research to understand how and when HTA transfers, as well as other aHTA approaches, are appropriate and helpful in LMICs, and what type of guidance is relevant for settings with fewer capacities, resources, and defined processes for HTA.

Strengths and limitations

The main strengths of this scoping review are the comprehensive search and up-to-date overview of tools, checklist, and approaches that can be used when considering an HTA transfer. We also addressed key issues related to definitions and missing guidance which may interfere with HTA transfers, specifically in LMIC settings.

This scoping review also has some limitations. Firstly, we focused on tools relevant to HTA analysis rather than the whole HTA process. Consequently, possible transfer tools related to “transferability” such as MCDA, deliberative processes for priority setting, appraisal, and frameworks for topic selection or stakeholder engagement that could be part of the transferability toolbox were outside of the scope of this review and excluded. Secondly, to reduce the volume of literature, the search strategy was focused on the clinical effectiveness and economic domains, and less explicitly on other HTA domains. Transferring existing evidence is complex, as it addresses many components and disciplines. The heterogeneity of the included studies reflects this complexity and made it difficult to generalize findings. We are aware that other relevant tools (e.g., country guidelines for rapid HTA methods, appraisal checklist, etc.) exist. Nevertheless, by including conference abstracts and the questionnaire to INAHTA members, we are confident that we have captured the most used transferability tools for HTA analysis currently available.

Conclusion

There is an increased interest in rapid and pragmatic HTA methods that, without using a de novo process, can answer policy questions rapidly, reduce duplication efforts, and make better use of limited resources. This scoping review identified tools from the published literature that can aid the transferability of HTA products. The EUnetHTA Adaptation Toolkit is the most comprehensive as it covers more HTA domains than the other tools. However, depending on the evidence, other checklists might be more appropriate. None of the identified tools covered all domains of an HTA, nor tackled all aspects of the transferability issues that we identified. Harmonization in HTA products and evidence-transfer processes, as well as pragmatic guidance for HTA transferability, especially for settings with limited HTA capacity, seem particularly warranted.