Background
The European network for health technology assessment (EUnetHTA) Joint Action 3 (JA3) aimed to develop a sustainable European model for future collaboration on HTA, by reducing duplication in HTA production and increasing patient access to health technologies. The production of EUnetHTA relative effectiveness assessments (REAs) on both pharmaceutical products and other technologies (e.g., medical devices and procedures) contributed to this.
The EUnetHTA REAs describe and summarize the available data and evidence of the health technology under assessment on (i) the health problem and current use of the technology; (ii) the technical characteristics of the technology; (iii) safety; and (iv) clinical effectiveness, compared to other existing alternatives (1). These four clinical domains are considered generalizable across countries. The EUnetHTA HTA Core Model®, which is a methodological framework for the production and sharing of HTA information, also includes domains on economic evaluation, ethical, organizational, patient/social, and legal aspects of the technology under assessment. These are, however, context dependent and should be considered on a national level (Reference Kristensen, Lampe and Wild2;Reference Kleijnen, Toenders and de Groot3).
This article describes the process and learnings for producing both pharmaceutical and other-technology REAs. Since the character of the technologies is different, some structural and procedural differences exist. Pharmaceutical REAs are centrally coordinated, based on a dossier submission by the health technology developer (HTD), focus on a single technology seeking Marketing Authorisation, and thus have an REA production timeline that is closely linked to the European regulatory pathway. Other technology REAs are coordinated either centrally or in a decentralized manner by so-called Activity Centre Department Leads (i.e., by selected EUnetHTA member agencies across Europe), focus on a single technology or class of products/technologies, and are less restricted to particular time points of a regulatory pathway and could also represent a re-assessment following additional or new (postmarketing) evidence (Reference Erdos, Ettinger, Mayer-Ferbas, de Villiers and Wild4). Given that a EUnetHTA REA solely provides a clinical assessment, its impact is to be considered at a national HTA production level and not the decision-making level.
In total, forty-three REAs (sixteen on pharmaceuticals and twenty-seven on other technologies) were produced in JA3. This REA production is built upon procedures, tools, templates, and guidelines developed in EUnetHTA Joint Action 2 (JA2). Extensive procedural changes, made based on experiences from JA2 (5) and JA3, were mapped in a paper with extensive recommendations for a future REA production process (6). This paper informed the development of the Future Model of Collaboration White Paper, which holds recommendations for a future European system on HTA collaboration (7;Reference Garrett, Imaz-Iglesia and Willamsen8).
This article presents frequently discussed aspects that led to procedural changes in JA3, aiming to enhance usability, transparency, and inclusiveness. For a full list of procedural changes made in JA3, please see Supplementary Table S1.
Methods
At the start of JA3, the main source of information was the JA2 technical report (5), which outlined findings, challenges, and recommendations for production processes. Additionally, a Quality Management System (QMS) was developed for the REA production in JA3 (Reference Luhnen, Ormstad and Willemsen9). For all components of the QMS and the refinement of the production procedures, the “Plan-Do-Check-Act-cycle" was applied (Reference Luhnen, Ormstad and Willemsen9). The main source of this evaluation was an online feedback survey after finalization of the REA (Supplementary File S2), to monitor the REA team’s (including the project manager) experiences on all aspects of the production process. Additionally, for pharmaceutical REAs three feedback workshops were held with the REA authors. For other technology REAs, the decentralized project managers provided input via email exchange with central project managers and via eleven meetings that focused on updates, discussions, and training. All feedback helped to identify areas for improvement, and REA teams and project managers were involved in making changes to counteract experienced challenges in the REA production process. During JA3, discussions on the REA production evolved as the frequency of topics mentioned changed after respective changes were initiated.
Additionally, the continuous EUnetHTA implementation surveys captured the use and non-use of the REAs in different European countries, and reasons thereof (10). EUnetHTA partners were asked to complete the implementation surveys after publication of each EUnetHTA REA and to update their survey entry in case of changes (10;11). No additional implementation analysis was done in the context of this manuscript.
All changes made to the EUnetHTA REA production process during JA3 were mapped, linked to pharmaceutical and/or other technologies, and were allocated to a period in which they were applied (start, mid, or end of JA3; see Table 1). Table 1 shows these changes and indicates which aspects (usability, transparency, and inclusiveness) were aimed to be positively influenced by the change.
Table 1. Overview of Measures Taken with the Purpose of Enhancing Usability, Transparency, and Inclusiveness of REAs
Results
EUnetHTA REAs and their use in national HTA production
Due to different HTA systems in Europe, different requirements for HTA procedures exist (Reference Chamova12). This also seems to impact how HTA agencies can use the EUnetHTA REAs in a national context. The EUnetHTA JA3 Implementation Report (2020) revealed various types of use of EUnetHTA REAs (10). Compared to JA2, the use increased for pharmaceutical REAs. In JA3, pharmaceutical REAs were more likely to support (national) HTA work, whereas other technology REAs replaced agency work in a majority of cases (10). For a EUnetHTA REA to be useful and informative to national and regional decision-making, it should fit as many information needs as possible.
Specifically, requirements on the use and the type of an evidence grading system are heterogeneous across Europe. Grading systems have become an important aspect of conducting evidence synthesis and the “Grading of Recommendations Assessment, Development and Evaluation”/GRADE-system appears to be the most commonly used (13). It is critical for EUnetHTA REAs that presentation of results and conclusions follows a standardized and predictable manner, but it should not interfere with national decision-making (Reference Luhnen, Ormstad and Willemsen9). Therefore, EUnetHTA described how to use GRADE partially in the EUnetHTA context to allow flexibility and adaptability for national assessors in using the information provided by the EUnetHTA REA authors (14).
Scoping of a specific EUnetHTA REA
Topic identification, prioritization, and selection
Key to ensure usability and impact of a EUnetHTA REA is selecting health technologies relevant to the HTA community (5;10). EUnetHTA defined recommendations for a Topic Identification, Selection, and Prioritisation (TISP) process, by investigating the needs for such process and piloting workflows. This resulted in Prioritisation Lists for pharmaceuticals and other technologies to facilitate acquisition of relevant technologies for REAs (15). However, challenges arose mainly with the timing of selecting technologies and ensuring their eligibility for a EUnetHTA REA.
Experiences around acquisition of relevant pharmaceutical compounds revealed that the expected level of REA re-use by other HTA agencies impacted the HTD decision to submit a compound for a EUnetHTA REA. Although the TISP process was piloted in JA3, no valid strategy could be established for other technology REAs due to the absence of a central database of CE-marked products (which might be available in 2022) and topics were mainly identified from partners’ work programs.
Defining the research question(s) in EUnetHTA REAs
To remove the need for a national clinical assessment when a EUnetHTA REA exists on the same technology, it is crucial to have a PICO (Population, Intervention, Comparators, Outcomes) in place that matches the requirements of as many HTA agencies as possible (6;Reference Luhnen, Ormstad and Willemsen9;10). EUnetHTA defined the role of a PICO as follows: the PICO should not be data driven, but should be based on the policy questions of HTA agencies; the PICO should inform the data requirements and the framework for the assessment; and the conclusions and evidence gaps should be drawn according to the PICO.
A PICO-survey was added to the production process of pharmaceutical REAs to allow EUnetHTA partners to contribute to defining the REA scope. For other technology REAs the optional PICO survey was not properly piloted, as the change was implemented at the end of JA3, therefore no data is available. For pharmaceutical REAs, the PICO survey was used thirteen times as of July 2018, to which on average fifteen HTA agencies contributed. On average for all conducted PICO surveys, there was 75 percent agreement on the proposed population, 88 percent on the proposed intervention, 59 percent on the proposed comparator(s) and 54 percent on the proposed outcome(s).
The number of changes per EUnetHTA REA and per PICO element (e.g., adding or removing subgroups/comparator(s)/outcome(s), and restructuring of the outcomes), based on the PICO survey and the Committee for Medicinal Products for Human Use (CHMP) opinion, shown in Figure 1, confirm the added value of a PICO survey in pharmaceutical REAs for defining a consolidated EUnetHTA PICO-question.
Figure 1. Number of changes made to the pharmaceutical REA PICO question based on the survey input. PTJAXX, assessment identification of pharmaceutical REAs.
EUnetHTA REA report templates
An important factor facilitating efficient production of a EUnetHTA REA, and influencing their use are the templates applied (10). For pharmaceutical REAs, all templates for the REA production were updated following JA2 experiences and a survey conducted in JA3 (5;16). Updates aimed to facilitate the provision of required data by HTDs, and to enhance usability by the REA teams and the end-users of the REAs (national HTA agencies) (Reference Luhnen, Ormstad and Willemsen9).
The REA report template for other technologies was revised based on feedback from REA teams. Further guidance on methodological and procedural issues, which was previously in separate documents, as well as information from the EUnetHTA HTA Core Model®, was added to the template (Reference Kristensen, Lampe and Wild2). The clear structure was welcomed by REA teams (Reference Luhnen, Ormstad and Willemsen9).
Furthermore, plain language summaries were piloted and a template was created during JA3. The summary enhances accessibility of the REA results to a wider public including patients, Healthcare Professionals (HCPs), policy makers, and other non-HTA experts.
Conflict of Interest and confidentiality
To ensure confidentiality and independence of the EUnetHTA REA team and the involved stakeholders, each participating individual signed a Declaration of Interest (DOI) form and Confidentiality Agreement prior to their involvement. To consistently evaluate and assess these forms, a Conflict of Interest (COI) Committee was established in JA3. This Committee followed a publicly available procedure guidance, which presents applied processes, precautions taken, and possible inclusion restrictions and critical COI situations (17). Mostly, critical situations leading to exclusion of an individual happened with HCP. It very rarely occurred with REA team members and never with patients or patient representatives. A General Data Protection Regulation (GDPR) compliant database accessible to a limited number of persons within EUnetHTA ensured secure storage of COI information (17). Avoiding involvement of individuals with a (major) COI and transparently outlining the applied rules and procedures, enables usability of the REA on national level and improves trust that the REA was not influenced by a COI (10).
Stakeholder interaction
Within a EUnetHTA REA, all relevant perspectives and expertise for the specific topic should be considered to ensure relevance and usability of REAs to HTA bodies (10;Reference Elvsaas, Ettinger and Willemsen18). HCPs can provide clarifications and important information on clinical pathways and course of diseases. Patients, patient representatives, and patient organizations (hereafter referred to as patients) or caregivers can offer valuable insights into the disease, daily life, and their perspectives on different treatments.
Methods for interaction with patients and HCPs
During JA3, recommended HCP involvement methods were refined (19). While in JA2, HCPs mainly participated in the review of draft documents (project plan and REA report), in JA3 further approaches were applied such as participation in the scoping e-meeting (in nine other technology REAs), and ad hoc contact during the scoping and assessment phase in case of upcoming questions (offered in the other technology REAs and applied in ten pharmaceutical REAs). In all other technology REAs and in two pharmaceutical REAs, HCPs conducted a review of the draft PICO and/or draft project plan and draft REA report. Dedicated standardized operating procedures (SOPs) for HCP involvement and checklists for HCPs were established and used.
Also, patient involvement was greatly improved and increased in JA3. Recommendations for involvement processes, based on extensive piloting, were published (20). Preliminary results of the experiences gathered with the different involvement methods applied, and efforts made to facilitate patient input in JA3 were presented in a special edition on patient involvement (Reference Elvsaas, Ettinger and Willemsen18). Patients were involved in fourteen pharmaceutical and twelve other technology REAs. The majority of involvement methods applied were the use of (online) patient input forms and one-on-one conversations.
Interaction with HTD
Major efforts were made to make the processes for interaction with HTDs transparent, procedurally fair, and clear on submission expectations. Procedure manuals, frequently asked questions (FAQ), dedicated contact details, and short videos were published on the EUnetHTA website (see Figure 2). The procedure manuals explained the processes, timelines, interaction with EUnetHTA, and the documents and tools that are used by the REA team (including guidance for a factual accuracy check by HTD), as well as the requirements for participation.
Table 2 outlines in which REA process step the HTD was involved and in which way. Due to the different nature of pharmaceuticals and other technologies, differences in HTD participation exist.
Table 2. Procedures for Involvement of HTD in EUnetHTA REAs
Accessibility of all relevant data to the end-users (HTA agencies) is a prerequisite for a transparent and unbiased REA, and is critical to the usability of an REA. Therefore, EUnetHTA developed submission requirements, outlining that all submitted information can be used by the REA team without any redaction prior to publication. However, the pharmaceutical industry challenged EUnetHTA’s citation and publication policy as they feared this may negatively impact the possibility of HTDs to publish their data and analysis in peer-reviewed journals (Reference Wilsdon, Pistollato and Damato21). In response, EUnetHTA stressed in an open letter to editors that having access to these data and analysis is a fundamental part of the REAs, facilitates perceived quality of the REA, and ultimately serves a public health interest (Reference Hedberg22).
In JA3 a process was applied where HTDs could do an optional factual accuracy check of the technical information presented in the REA report (n = 14 for pharmaceutical REAs and n = 23 for other technology REAs). In other technologies, this was also offered for the draft project plan (n = 15), where detailed information on the products considered as an intervention in the REA was included. Comments provided by the HTDs were answered by the REA team and published together with the final documents (project plan or REA report).
Transparency of methods and procedures
In JA3, more emphasis was put on disseminating information to announce the REA start (immediately when the team was known), as well as the scope and timelines of an REA. In contrast to JA2, the project plan was published on the EUnetHTA website as soon as it was final (for pharmaceutical REAs this means immediate publication after positive CHMP opinion is adopted), to ensure transparency on all relevant aspects (PICO, stakeholder involvement, timelines) prior to the REA start (10). In case of timeline deviations, these were amended on the website and in the project plan.
As indicated in the submission requirements documents the (core)-submission dossier (if available) was published to make the sources that were used in the REA production available. In other technology REAs, this was implemented at the end of JA3.
To facilitate transparency, all relevant information for HTA agencies and external stakeholders are stored on the EUnetHTA website, with an easy navigation (see Figure 2). The headers relevant for the REA production are
-
– “Assessments”: leading to, for example, the ongoing and published REAs, a FAQ about the REA production process and information about national uptake;
-
– “Tools”: leading to, for example, the methodological guidelines and recommendations, templates, DOI information, and
-
– “Get Involved": leading to information for all external stakeholders for example, on how they can participate in an REA, and ongoing online patient consultation.
Timely availability of EUnetHTA REAs
Some EUnetHTA partners required the other technology REA reports prior to the publication due to their national deadlines (10). In response, a process was adopted to facilitate this, as this should prevent duplication of HTA production. For pharmaceutical REAs, HTA agencies often start their national assessment after the compound has received marketing authorization by the European Medicines Agency (EMA) (11). In JA3, the final pharmaceutical REA was published on average 22 days after European public assessment report (EPAR) availability, compared to 108 days in JA2. This major improvement was achieved by requesting the HTD Submission Dossier prior to positive CHMP opinion and by facilitating timely information to regulatory output (under respective remits and confidentiality) to prevent duplication with EPAR (10).
Project management—the importance of the role in supporting usability, transparency and inclusiveness
The EUnetHTA REAs project managers were responsible for coordinating the REA production, monitoring the production timelines, and ensuring awareness of EUnetHTA guidelines and SOPs. Furthermore, they were the primary contact point for internal and external communication. Only in dedicated situations, the communication could be shifted to the REA author if this would enhance efficiency of the REA production. The project management was fine-tuned and further standardized in JA3.
An important finding from the feedback surveys was that the authoring teams of the REAs felt the project manager role is a separate, well defined, and important role for REA coordination and production. A critical aspect to enable project managers to operate independently and in a standard manner was the availability of the EUnetHTA Companion Guide. The Companion Guide, only accessible for EUnetHTA partners, stores all SOPs, procedures, guidances, and templates to be used for an REA production (Reference Luhnen, Ormstad and Willemsen9).
Discussion
Compared to JA2, the EUnetHTA REA production increased during JA3. The aim of JA3 was to develop a sustainable model for European collaboration on HTA. The REA production plays a significant role in this. While JA3 aimed to reach a maximum impact of the produced REAs, there was still a focus on piloting production blueprints and collaboration models in an attempt to support the development of a sustainable EU collaboration model. Therefore, continuous procedural changes to the REA production process and procedures were made and the quality management system was fine-tuned, always with the aim to maximize usability, inclusiveness, and transparency.
The procedural changes (see Table 1) were based on qualitative (e.g., JA2 experiences and recommendations and JA3 feedback workshops) and quantitative (i.e., evaluation and implementation surveys) feedback. The usability of EUnetHTA REAs was aimed to be improved by focussing on the needs of the EUnetHTA REA producers as well as users (HTA agencies). Huge steps were taken with regard to transparency, which was achieved through the publication of guidances, templates, and up-to-date information on the EUnetHTA website. The inclusiveness was improved due to enhanced and streamlined (stakeholder) interaction and involvement as well as feedback procedures. The fine-tuned project management brought all aspects together and ensured a consistent and reliable workflow.
Kleijnen et al. (Reference Kleijnen, George and Goulden23) found that although similarities exist between HTA agencies in the choice of comparators, differences occur in the preferred outcomes. In line with this, the PICO-survey analysis showed substantial differences between HTA agencies on inclusion of relevant outcomes and comparators. Therefore, the PICO survey was welcomed by EUnetHTA partners, especially by the REA authors as it facilitated the creation of a relevant PICO-question, and eventually an REA report, for many partners. However, it is challenging to balance the individual HTA agencies’ needs with the feasibility to conduct the REA in a limited timeframe. Including all possible information needs in the REA could hinder the feasibility of a timely REA production and overburden HTA agencies in implementing the REA findings in the national and/or regional decision-making (Reference Chamova12). Regardless, the PICO survey could be a helpful tool to encourage adoption of the EUnetHTA REA without any changes as it ensures the REA is more relevant (10).
This article only describes changes made within JA3, however, also national HTA agencies and stakeholders (such as submitting HTDs) may need to adapt their procedures to be able to comply with the EUnetHTA REA process. Harrison et al. (Reference Harrison, Fischer and Walpola24) stated that internal and external stakeholders are more likely to embrace a certain change if they are directly or indirectly affected by it. We experienced this, as EUnetHTA members that we communicated with or that actively contributed to a REA, became familiar with the REA procedures and might have adjusted their internal procedures and communication to ensure timely response to the production steps and to possibly enhance uptake of an REA (10).
Due to a number of limitations, such as no additional analysis of the effect of procedural changes on the implementation of REAs; different ways of measuring usage and barriers of usage in JA2 and JA3, it is challenging to reach strong conclusions on the impact the procedural changes have made. However, there are strong signals that the procedural changes positively influenced the impact of the REAs were received during feedback workshops and surveys, as well as the implementation surveys conducted throughout JA3.
No in-depth analysis with regard to the implementation or uptake of EUnetHTA REAs on a national level was conducted, as to capture a complete level of REA implementation, 18 months to 2 years of follow-up for pharmaceutical REAs and approximately 3 years for other technology REAs would be required. Especially for other technology REAs, the relevance of topics is strongly affected by the fact that technologies can be introduced in different countries at different time points. However, the present implementation reports were consulted to see trends on how the use of EUnetHTA REAs and reasons for non-use looked like at a specific point in time. There was no extensive collaboration with colleagues working on the implementation tasks, only the results were considered to get some insights in how to improve the production processes.
Another limitation of this article is that the feedback collection lied on HTA agencies (EUnetHTA JA3 members), which produced EUnetHTA REAs and/or used them on a national level. However, to improve procedural aspects related to external stakeholders, that is, HTD, patients, and HCP, experiences of involved stakeholders should also be analyzed.
Although discussions on the REA production evolved during JA3, some issues were reoccurring and could not have been finally solved completely. It is important that these issues are addressed in EUnetHTA 21, the follow-up project tasked to supporting the further development of guidance documents and/or drafting of implementing legislation required within the future HTA regulation. Whilst the changes made in JA3 seem to have positively influenced the usability, transparency, and inclusiveness of the REAs, EUnetHTA JA3 was a voluntary project and could not solve all perceived challenges. The proposed future European HTA regulation is expected to further inform the collaborative procedures for developing joint clinical REAs and might provide the legislative mandate to solve the remaining challenges (25). It is of utmost importance that the experiences and recommendations of JA3 REA production are considered in the new service contract EUnetHTA 21 to support implementation of the future HTA regulation.
Conclusion/Summary
The continuous changes made to the production process aiming to increase usability, transparency, and inclusiveness, contributed to the objective of defining a sustainable model for future collaboration on HTA. Despite that HTA agencies have different national HTA requirements, the procedural changes made to the JA3 REA production facilitated easy and consistent production processes, but also aimed to support the use of REAs by HTA agencies at a national level. The goal was to ensure predictability, high-quality standards, and usability of the REA reports. Nevertheless, it is of utmost importance that further perceived differences on a methodological level are being solved to ensure a strong base for future European collaboration on REA production.
Open access
Background
The European network for health technology assessment (EUnetHTA) Joint Action 3 (JA3) aimed to develop a sustainable European model for future collaboration on HTA, by reducing duplication in HTA production and increasing patient access to health technologies. The production of EUnetHTA relative effectiveness assessments (REAs) on both pharmaceutical products and other technologies (e.g., medical devices and procedures) contributed to this.
The EUnetHTA REAs describe and summarize the available data and evidence of the health technology under assessment on (i) the health problem and current use of the technology; (ii) the technical characteristics of the technology; (iii) safety; and (iv) clinical effectiveness, compared to other existing alternatives (1). These four clinical domains are considered generalizable across countries. The EUnetHTA HTA Core Model®, which is a methodological framework for the production and sharing of HTA information, also includes domains on economic evaluation, ethical, organizational, patient/social, and legal aspects of the technology under assessment. These are, however, context dependent and should be considered on a national level (Reference Kristensen, Lampe and Wild2;Reference Kleijnen, Toenders and de Groot3).
This article describes the process and learnings for producing both pharmaceutical and other-technology REAs. Since the character of the technologies is different, some structural and procedural differences exist. Pharmaceutical REAs are centrally coordinated, based on a dossier submission by the health technology developer (HTD), focus on a single technology seeking Marketing Authorisation, and thus have an REA production timeline that is closely linked to the European regulatory pathway. Other technology REAs are coordinated either centrally or in a decentralized manner by so-called Activity Centre Department Leads (i.e., by selected EUnetHTA member agencies across Europe), focus on a single technology or class of products/technologies, and are less restricted to particular time points of a regulatory pathway and could also represent a re-assessment following additional or new (postmarketing) evidence (Reference Erdos, Ettinger, Mayer-Ferbas, de Villiers and Wild4). Given that a EUnetHTA REA solely provides a clinical assessment, its impact is to be considered at a national HTA production level and not the decision-making level.
In total, forty-three REAs (sixteen on pharmaceuticals and twenty-seven on other technologies) were produced in JA3. This REA production is built upon procedures, tools, templates, and guidelines developed in EUnetHTA Joint Action 2 (JA2). Extensive procedural changes, made based on experiences from JA2 (5) and JA3, were mapped in a paper with extensive recommendations for a future REA production process (6). This paper informed the development of the Future Model of Collaboration White Paper, which holds recommendations for a future European system on HTA collaboration (7;Reference Garrett, Imaz-Iglesia and Willamsen8).
This article presents frequently discussed aspects that led to procedural changes in JA3, aiming to enhance usability, transparency, and inclusiveness. For a full list of procedural changes made in JA3, please see Supplementary Table S1.
Methods
At the start of JA3, the main source of information was the JA2 technical report (5), which outlined findings, challenges, and recommendations for production processes. Additionally, a Quality Management System (QMS) was developed for the REA production in JA3 (Reference Luhnen, Ormstad and Willemsen9). For all components of the QMS and the refinement of the production procedures, the “Plan-Do-Check-Act-cycle" was applied (Reference Luhnen, Ormstad and Willemsen9). The main source of this evaluation was an online feedback survey after finalization of the REA (Supplementary File S2), to monitor the REA team’s (including the project manager) experiences on all aspects of the production process. Additionally, for pharmaceutical REAs three feedback workshops were held with the REA authors. For other technology REAs, the decentralized project managers provided input via email exchange with central project managers and via eleven meetings that focused on updates, discussions, and training. All feedback helped to identify areas for improvement, and REA teams and project managers were involved in making changes to counteract experienced challenges in the REA production process. During JA3, discussions on the REA production evolved as the frequency of topics mentioned changed after respective changes were initiated.
Additionally, the continuous EUnetHTA implementation surveys captured the use and non-use of the REAs in different European countries, and reasons thereof (10). EUnetHTA partners were asked to complete the implementation surveys after publication of each EUnetHTA REA and to update their survey entry in case of changes (10;11). No additional implementation analysis was done in the context of this manuscript.
All changes made to the EUnetHTA REA production process during JA3 were mapped, linked to pharmaceutical and/or other technologies, and were allocated to a period in which they were applied (start, mid, or end of JA3; see Table 1). Table 1 shows these changes and indicates which aspects (usability, transparency, and inclusiveness) were aimed to be positively influenced by the change.
Table 1. Overview of Measures Taken with the Purpose of Enhancing Usability, Transparency, and Inclusiveness of REAs
COI, conflict of interest; EPAR, European public assessment report; EUnetHTA, European network for health technology assessment; HCP, health care professionals; HTD, health technology developer; OT, other technology REA; PICO, population, intervention, comparator and outcomes; PT, pharmaceutical REA; REA, relative effectiveness assessment; SOP, standardized operating procedure.
Note. Start JA3 = 2016/2017, Mid JA3 = 2018/2019, End JA3 = 2020/2021. This indication—for which type of REA which aspect (usability, transparency, and inclusiveness) was aimed to be positively influenced by the change—is based on the authors’ own interpretation, and it is not based on any further analysis or research.
Results
EUnetHTA REAs and their use in national HTA production
Due to different HTA systems in Europe, different requirements for HTA procedures exist (Reference Chamova12). This also seems to impact how HTA agencies can use the EUnetHTA REAs in a national context. The EUnetHTA JA3 Implementation Report (2020) revealed various types of use of EUnetHTA REAs (10). Compared to JA2, the use increased for pharmaceutical REAs. In JA3, pharmaceutical REAs were more likely to support (national) HTA work, whereas other technology REAs replaced agency work in a majority of cases (10). For a EUnetHTA REA to be useful and informative to national and regional decision-making, it should fit as many information needs as possible.
Specifically, requirements on the use and the type of an evidence grading system are heterogeneous across Europe. Grading systems have become an important aspect of conducting evidence synthesis and the “Grading of Recommendations Assessment, Development and Evaluation”/GRADE-system appears to be the most commonly used (13). It is critical for EUnetHTA REAs that presentation of results and conclusions follows a standardized and predictable manner, but it should not interfere with national decision-making (Reference Luhnen, Ormstad and Willemsen9). Therefore, EUnetHTA described how to use GRADE partially in the EUnetHTA context to allow flexibility and adaptability for national assessors in using the information provided by the EUnetHTA REA authors (14).
Scoping of a specific EUnetHTA REA
Topic identification, prioritization, and selection
Key to ensure usability and impact of a EUnetHTA REA is selecting health technologies relevant to the HTA community (5;10). EUnetHTA defined recommendations for a Topic Identification, Selection, and Prioritisation (TISP) process, by investigating the needs for such process and piloting workflows. This resulted in Prioritisation Lists for pharmaceuticals and other technologies to facilitate acquisition of relevant technologies for REAs (15). However, challenges arose mainly with the timing of selecting technologies and ensuring their eligibility for a EUnetHTA REA.
Experiences around acquisition of relevant pharmaceutical compounds revealed that the expected level of REA re-use by other HTA agencies impacted the HTD decision to submit a compound for a EUnetHTA REA. Although the TISP process was piloted in JA3, no valid strategy could be established for other technology REAs due to the absence of a central database of CE-marked products (which might be available in 2022) and topics were mainly identified from partners’ work programs.
Defining the research question(s) in EUnetHTA REAs
To remove the need for a national clinical assessment when a EUnetHTA REA exists on the same technology, it is crucial to have a PICO (Population, Intervention, Comparators, Outcomes) in place that matches the requirements of as many HTA agencies as possible (6;Reference Luhnen, Ormstad and Willemsen9;10). EUnetHTA defined the role of a PICO as follows: the PICO should not be data driven, but should be based on the policy questions of HTA agencies; the PICO should inform the data requirements and the framework for the assessment; and the conclusions and evidence gaps should be drawn according to the PICO.
A PICO-survey was added to the production process of pharmaceutical REAs to allow EUnetHTA partners to contribute to defining the REA scope. For other technology REAs the optional PICO survey was not properly piloted, as the change was implemented at the end of JA3, therefore no data is available. For pharmaceutical REAs, the PICO survey was used thirteen times as of July 2018, to which on average fifteen HTA agencies contributed. On average for all conducted PICO surveys, there was 75 percent agreement on the proposed population, 88 percent on the proposed intervention, 59 percent on the proposed comparator(s) and 54 percent on the proposed outcome(s).
The number of changes per EUnetHTA REA and per PICO element (e.g., adding or removing subgroups/comparator(s)/outcome(s), and restructuring of the outcomes), based on the PICO survey and the Committee for Medicinal Products for Human Use (CHMP) opinion, shown in Figure 1, confirm the added value of a PICO survey in pharmaceutical REAs for defining a consolidated EUnetHTA PICO-question.
Figure 1. Number of changes made to the pharmaceutical REA PICO question based on the survey input. PTJAXX, assessment identification of pharmaceutical REAs.
EUnetHTA REA report templates
An important factor facilitating efficient production of a EUnetHTA REA, and influencing their use are the templates applied (10). For pharmaceutical REAs, all templates for the REA production were updated following JA2 experiences and a survey conducted in JA3 (5;16). Updates aimed to facilitate the provision of required data by HTDs, and to enhance usability by the REA teams and the end-users of the REAs (national HTA agencies) (Reference Luhnen, Ormstad and Willemsen9).
The REA report template for other technologies was revised based on feedback from REA teams. Further guidance on methodological and procedural issues, which was previously in separate documents, as well as information from the EUnetHTA HTA Core Model®, was added to the template (Reference Kristensen, Lampe and Wild2). The clear structure was welcomed by REA teams (Reference Luhnen, Ormstad and Willemsen9).
Furthermore, plain language summaries were piloted and a template was created during JA3. The summary enhances accessibility of the REA results to a wider public including patients, Healthcare Professionals (HCPs), policy makers, and other non-HTA experts.
Conflict of Interest and confidentiality
To ensure confidentiality and independence of the EUnetHTA REA team and the involved stakeholders, each participating individual signed a Declaration of Interest (DOI) form and Confidentiality Agreement prior to their involvement. To consistently evaluate and assess these forms, a Conflict of Interest (COI) Committee was established in JA3. This Committee followed a publicly available procedure guidance, which presents applied processes, precautions taken, and possible inclusion restrictions and critical COI situations (17). Mostly, critical situations leading to exclusion of an individual happened with HCP. It very rarely occurred with REA team members and never with patients or patient representatives. A General Data Protection Regulation (GDPR) compliant database accessible to a limited number of persons within EUnetHTA ensured secure storage of COI information (17). Avoiding involvement of individuals with a (major) COI and transparently outlining the applied rules and procedures, enables usability of the REA on national level and improves trust that the REA was not influenced by a COI (10).
Stakeholder interaction
Within a EUnetHTA REA, all relevant perspectives and expertise for the specific topic should be considered to ensure relevance and usability of REAs to HTA bodies (10;Reference Elvsaas, Ettinger and Willemsen18). HCPs can provide clarifications and important information on clinical pathways and course of diseases. Patients, patient representatives, and patient organizations (hereafter referred to as patients) or caregivers can offer valuable insights into the disease, daily life, and their perspectives on different treatments.
Methods for interaction with patients and HCPs
During JA3, recommended HCP involvement methods were refined (19). While in JA2, HCPs mainly participated in the review of draft documents (project plan and REA report), in JA3 further approaches were applied such as participation in the scoping e-meeting (in nine other technology REAs), and ad hoc contact during the scoping and assessment phase in case of upcoming questions (offered in the other technology REAs and applied in ten pharmaceutical REAs). In all other technology REAs and in two pharmaceutical REAs, HCPs conducted a review of the draft PICO and/or draft project plan and draft REA report. Dedicated standardized operating procedures (SOPs) for HCP involvement and checklists for HCPs were established and used.
Also, patient involvement was greatly improved and increased in JA3. Recommendations for involvement processes, based on extensive piloting, were published (20). Preliminary results of the experiences gathered with the different involvement methods applied, and efforts made to facilitate patient input in JA3 were presented in a special edition on patient involvement (Reference Elvsaas, Ettinger and Willemsen18). Patients were involved in fourteen pharmaceutical and twelve other technology REAs. The majority of involvement methods applied were the use of (online) patient input forms and one-on-one conversations.
Interaction with HTD
Major efforts were made to make the processes for interaction with HTDs transparent, procedurally fair, and clear on submission expectations. Procedure manuals, frequently asked questions (FAQ), dedicated contact details, and short videos were published on the EUnetHTA website (see Figure 2). The procedure manuals explained the processes, timelines, interaction with EUnetHTA, and the documents and tools that are used by the REA team (including guidance for a factual accuracy check by HTD), as well as the requirements for participation.
Figure 2. Overview of the website navigation—www.eunethta.eu.
Table 2 outlines in which REA process step the HTD was involved and in which way. Due to the different nature of pharmaceuticals and other technologies, differences in HTD participation exist.
Table 2. Procedures for Involvement of HTD in EUnetHTA REAs
EUnetHTA, European network for health technology assessment; HTD, health technology developer; PICO, population, intervention, comparator and outcomes; REA, relative effectiveness assessment.
Accessibility of all relevant data to the end-users (HTA agencies) is a prerequisite for a transparent and unbiased REA, and is critical to the usability of an REA. Therefore, EUnetHTA developed submission requirements, outlining that all submitted information can be used by the REA team without any redaction prior to publication. However, the pharmaceutical industry challenged EUnetHTA’s citation and publication policy as they feared this may negatively impact the possibility of HTDs to publish their data and analysis in peer-reviewed journals (Reference Wilsdon, Pistollato and Damato21). In response, EUnetHTA stressed in an open letter to editors that having access to these data and analysis is a fundamental part of the REAs, facilitates perceived quality of the REA, and ultimately serves a public health interest (Reference Hedberg22).
In JA3 a process was applied where HTDs could do an optional factual accuracy check of the technical information presented in the REA report (n = 14 for pharmaceutical REAs and n = 23 for other technology REAs). In other technologies, this was also offered for the draft project plan (n = 15), where detailed information on the products considered as an intervention in the REA was included. Comments provided by the HTDs were answered by the REA team and published together with the final documents (project plan or REA report).
Transparency of methods and procedures
In JA3, more emphasis was put on disseminating information to announce the REA start (immediately when the team was known), as well as the scope and timelines of an REA. In contrast to JA2, the project plan was published on the EUnetHTA website as soon as it was final (for pharmaceutical REAs this means immediate publication after positive CHMP opinion is adopted), to ensure transparency on all relevant aspects (PICO, stakeholder involvement, timelines) prior to the REA start (10). In case of timeline deviations, these were amended on the website and in the project plan.
As indicated in the submission requirements documents the (core)-submission dossier (if available) was published to make the sources that were used in the REA production available. In other technology REAs, this was implemented at the end of JA3.
To facilitate transparency, all relevant information for HTA agencies and external stakeholders are stored on the EUnetHTA website, with an easy navigation (see Figure 2). The headers relevant for the REA production are
– “Assessments”: leading to, for example, the ongoing and published REAs, a FAQ about the REA production process and information about national uptake;
– “Tools”: leading to, for example, the methodological guidelines and recommendations, templates, DOI information, and
– “Get Involved": leading to information for all external stakeholders for example, on how they can participate in an REA, and ongoing online patient consultation.
Timely availability of EUnetHTA REAs
Some EUnetHTA partners required the other technology REA reports prior to the publication due to their national deadlines (10). In response, a process was adopted to facilitate this, as this should prevent duplication of HTA production. For pharmaceutical REAs, HTA agencies often start their national assessment after the compound has received marketing authorization by the European Medicines Agency (EMA) (11). In JA3, the final pharmaceutical REA was published on average 22 days after European public assessment report (EPAR) availability, compared to 108 days in JA2. This major improvement was achieved by requesting the HTD Submission Dossier prior to positive CHMP opinion and by facilitating timely information to regulatory output (under respective remits and confidentiality) to prevent duplication with EPAR (10).
Project management—the importance of the role in supporting usability, transparency and inclusiveness
The EUnetHTA REAs project managers were responsible for coordinating the REA production, monitoring the production timelines, and ensuring awareness of EUnetHTA guidelines and SOPs. Furthermore, they were the primary contact point for internal and external communication. Only in dedicated situations, the communication could be shifted to the REA author if this would enhance efficiency of the REA production. The project management was fine-tuned and further standardized in JA3.
An important finding from the feedback surveys was that the authoring teams of the REAs felt the project manager role is a separate, well defined, and important role for REA coordination and production. A critical aspect to enable project managers to operate independently and in a standard manner was the availability of the EUnetHTA Companion Guide. The Companion Guide, only accessible for EUnetHTA partners, stores all SOPs, procedures, guidances, and templates to be used for an REA production (Reference Luhnen, Ormstad and Willemsen9).
Discussion
Compared to JA2, the EUnetHTA REA production increased during JA3. The aim of JA3 was to develop a sustainable model for European collaboration on HTA. The REA production plays a significant role in this. While JA3 aimed to reach a maximum impact of the produced REAs, there was still a focus on piloting production blueprints and collaboration models in an attempt to support the development of a sustainable EU collaboration model. Therefore, continuous procedural changes to the REA production process and procedures were made and the quality management system was fine-tuned, always with the aim to maximize usability, inclusiveness, and transparency.
The procedural changes (see Table 1) were based on qualitative (e.g., JA2 experiences and recommendations and JA3 feedback workshops) and quantitative (i.e., evaluation and implementation surveys) feedback. The usability of EUnetHTA REAs was aimed to be improved by focussing on the needs of the EUnetHTA REA producers as well as users (HTA agencies). Huge steps were taken with regard to transparency, which was achieved through the publication of guidances, templates, and up-to-date information on the EUnetHTA website. The inclusiveness was improved due to enhanced and streamlined (stakeholder) interaction and involvement as well as feedback procedures. The fine-tuned project management brought all aspects together and ensured a consistent and reliable workflow.
Kleijnen et al. (Reference Kleijnen, George and Goulden23) found that although similarities exist between HTA agencies in the choice of comparators, differences occur in the preferred outcomes. In line with this, the PICO-survey analysis showed substantial differences between HTA agencies on inclusion of relevant outcomes and comparators. Therefore, the PICO survey was welcomed by EUnetHTA partners, especially by the REA authors as it facilitated the creation of a relevant PICO-question, and eventually an REA report, for many partners. However, it is challenging to balance the individual HTA agencies’ needs with the feasibility to conduct the REA in a limited timeframe. Including all possible information needs in the REA could hinder the feasibility of a timely REA production and overburden HTA agencies in implementing the REA findings in the national and/or regional decision-making (Reference Chamova12). Regardless, the PICO survey could be a helpful tool to encourage adoption of the EUnetHTA REA without any changes as it ensures the REA is more relevant (10).
This article only describes changes made within JA3, however, also national HTA agencies and stakeholders (such as submitting HTDs) may need to adapt their procedures to be able to comply with the EUnetHTA REA process. Harrison et al. (Reference Harrison, Fischer and Walpola24) stated that internal and external stakeholders are more likely to embrace a certain change if they are directly or indirectly affected by it. We experienced this, as EUnetHTA members that we communicated with or that actively contributed to a REA, became familiar with the REA procedures and might have adjusted their internal procedures and communication to ensure timely response to the production steps and to possibly enhance uptake of an REA (10).
Due to a number of limitations, such as no additional analysis of the effect of procedural changes on the implementation of REAs; different ways of measuring usage and barriers of usage in JA2 and JA3, it is challenging to reach strong conclusions on the impact the procedural changes have made. However, there are strong signals that the procedural changes positively influenced the impact of the REAs were received during feedback workshops and surveys, as well as the implementation surveys conducted throughout JA3.
No in-depth analysis with regard to the implementation or uptake of EUnetHTA REAs on a national level was conducted, as to capture a complete level of REA implementation, 18 months to 2 years of follow-up for pharmaceutical REAs and approximately 3 years for other technology REAs would be required. Especially for other technology REAs, the relevance of topics is strongly affected by the fact that technologies can be introduced in different countries at different time points. However, the present implementation reports were consulted to see trends on how the use of EUnetHTA REAs and reasons for non-use looked like at a specific point in time. There was no extensive collaboration with colleagues working on the implementation tasks, only the results were considered to get some insights in how to improve the production processes.
Another limitation of this article is that the feedback collection lied on HTA agencies (EUnetHTA JA3 members), which produced EUnetHTA REAs and/or used them on a national level. However, to improve procedural aspects related to external stakeholders, that is, HTD, patients, and HCP, experiences of involved stakeholders should also be analyzed.
Although discussions on the REA production evolved during JA3, some issues were reoccurring and could not have been finally solved completely. It is important that these issues are addressed in EUnetHTA 21, the follow-up project tasked to supporting the further development of guidance documents and/or drafting of implementing legislation required within the future HTA regulation. Whilst the changes made in JA3 seem to have positively influenced the usability, transparency, and inclusiveness of the REAs, EUnetHTA JA3 was a voluntary project and could not solve all perceived challenges. The proposed future European HTA regulation is expected to further inform the collaborative procedures for developing joint clinical REAs and might provide the legislative mandate to solve the remaining challenges (25). It is of utmost importance that the experiences and recommendations of JA3 REA production are considered in the new service contract EUnetHTA 21 to support implementation of the future HTA regulation.
Conclusion/Summary
The continuous changes made to the production process aiming to increase usability, transparency, and inclusiveness, contributed to the objective of defining a sustainable model for future collaboration on HTA. Despite that HTA agencies have different national HTA requirements, the procedural changes made to the JA3 REA production facilitated easy and consistent production processes, but also aimed to support the use of REAs by HTA agencies at a national level. The goal was to ensure predictability, high-quality standards, and usability of the REA reports. Nevertheless, it is of utmost importance that further perceived differences on a methodological level are being solved to ensure a strong base for future European collaboration on REA production.
Supplementary Material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462322000058.
Funding Statement
The contents of this article arise from the project “724130/EUnetHTA JA3,” which has received funding from the EU, in the framework of the Health Programme (2014–2020). Sole responsibility for its contents lies with the author(s) and neither the EUnetHTA Coordinator, the European Commission, nor any other body of the European Union is responsible for any use that may be made of the information contained therein.
Conflicts of Interest
The authors declare that they have no competing interests.