International overview of health technology assessment training tools and materials for patients and consumers

Abstract

Objectives

The study, conducted by the French National Authority for Health (HAS), aimed to identify available online health technology assessment (HTA) training tools for patients, specifically those used by HTA bodies (HTAbs) and major selected European and international patient and consumer groups (PCGs), to inform an HTA training approach for HAS.

Methods

A literature search, a review of selected websites, semi-structured interviews with HTAbs and patient groups, and discussion within a dedicated working group to help target the needs and preferences of patients and consumers were conducted. Online HTA training tools relating to HTA and patient and public involvement (PPI), published or translated into English or French, were included in the study results.

Results

Eighty-two online HTA training tools for patients and consumers were selected according to the specified inclusion criteria coming from sixteen international HTAbs, nine European and international PCGs, and thirteen other organizations. Two main categories emerged: the first relating to HTA and the second relating to PPI in HTA. The formats of these tools ranged from interactive and non-interactive formats with varying accessibility and assessment methods. No journal articles mentioned explicitly the content and format of PCG training tools.

Conclusions

This research served as a basis for HAS to develop their own HTA training tools and materials for patients and consumers. Two training tools were subsequently developed, guided by the needs and preferences of a patient and consumer working group, and were published on the HAS website in November 2022.

Background

Health technology assessment (HTA) has evolved over the past 20 years, involving patients (including carers or patient representatives), consumers (the public) and patient and consumer groups (PCG) in the appraisal process, and providing valuable patient-based evidence (1; 2). Training and capacity building for these individuals and groups to contribute to HTA has become important resulting in the development of training tools by HTA bodies (HTAbs) or networks of HTAbs (3–7).

In 2017, The French National Authority for Health (HAS) established an open, online, systematic contribution process facilitating PCG contributions to French HTA (8). Following the establishment of this process and reinforced by one of the pillars of HAS Strategic Plan 2019–2024 whose aim is to make public involvement a priority at HAS (9), the Public Involvement Department (SEU) was created in 2019. This has enabled patient and consumer contributions to become increasingly integrated within the HTA process in France. Diverse participation methods for PCGs as stakeholders are possible including written contributions, hearings before an HTA committee, group interviews, public consultations, or reviewing documents prior to their publication (10). Patients and consumers are also able to collaborate with HAS as “individual experts” within committees, working groups, and via interviews where they are compensated for their time (10). These contributions are complementary to clinical and economic data used to guide decisions made by HAS’ committees who appraise the HTA report and in certain circumstances, the HAS Board who adopts the opinion before its transmission to the French Ministry of Health for a decision on pricing and reimbursement.

In 2020, workshops with PCGs were run by the SEU aiming to identify opportunities for improving public involvement in HTA activities at HAS. Bilateral interviews with several participants highlighted the growing demand by French PCGs to be trained to contribute more efficiently to HAS HTA processes, for example, the assessment of drugs, medical devices, and professional practices for reimbursement (11). Given the context of the COVID-19 pandemic, there was a specific demand for online training tools. Subsequently, the “Strengthening public involvement in HTA at HAS" roadmap was written in July 2021 (12). This roadmap includes a specific action to train PCGs for their contribution to HTA at HAS and is coupled with the elaboration of targeted online HTA training tools (12).

To complete this action of the roadmap and respond to the demands of the French PCGs, the project was designed to include several stages. The aim of the first stage was twofold. First, to conduct an international study through a review to identify online HTA training tools and to analyze and compare the tools currently offered. Although reviews have been conducted for other topics related to patient and public involvement (PPI) in HTA (13), a knowledge gap exists on this topic. An online HTA training tool or material (henceforth referred to as: training tool) is defined here as an instrument used to educate a public on HTA on how they are affected by and can contribute to HTA processes, delivered through an online platform. The study specifically examined those used by HTAbs outside of France and major French, European, and international PCGs (13). Second, to present the results to a patient and consumer working group to see if they would meet the needs and to identify other needs in terms of the themes and format of the HTA training required.

The second stage of the project aimed to develop a training tool corresponding to the patient and consumer working group’s needs to facilitate their contribution to HTA at HAS. Colleagues from across the HAS HTA division provided feedback throughout this stage.

This article focuses on the first stage of the project and the analysis/use of these results by the patient and consumer working group.

Methods

The methods used to identify training tools were a literature search, a review of websites, semi-structured interviews with selected HTAbs and PCGs, and a review of selected PCG websites. A discussion within the working group helped target the needs and preferences of patients and consumers.

Literature search

A systematic literature search including relevant articles between 1 January 2011 and 31 December 2021 was conducted in December 2021:

• • International literature: Medline Database and Cochrane Library Database.

• • French literature: Cismef, Lissa.

MeSH terms for the Medline database and the Cochrane Library were as follows:

• • Cochrane: “online education” OR ((training OR course) AND patient*) OR (e-learning AND patient*)’

• • Medline: “online education” OR (training course* AND patient*) OR ((training AND course* AND patient*) AND (online OR cyber OR web))

Review of the following websites:

• • Websites publishing recommendations and HTA reports, and health-related professional organizations listed on the websites of the International Network of Agencies for Health Technology Assessment (INAHTA), the European Network for HTA (EUnetHTA), Health Technology Assessment International (HTAi);

• • Websites identified through the European Patients’ Forum (EPF);

• • Websites providing HTA training for the public (e.g., European Patients’ Academy on Therapeutic Innovation (EUPATI) and Patient Voice Initiative, Australia),

• • The British National Institute of Health and Care Research (NIHR);

Each site was examined to find existing training tools (see Supplementary File 1). For each, a search was made using the keywords “health technology assessment” or “training” or “education” or “e-learning”.

Inclusion criteria

Studies/tools were selected in terms of the following inclusion criteria:

• • Source: Training tools from HTAbs, PCGs (i.e., none coming directly from the pharmaceutical industry), and other relevant bodies such as professional societies (e.g., HTAi)

• • Themes: All topics relating to HTA and PPI in HTA (e.g., what is HTA, how patients and consumers can contribute to the assessment of a medicine, medical device, professional practice, etc.)

• • Format: All online formats.

• • Language: French, English, or other languages with English or French translations or a summary provided by our contacts.

All training tools meeting the inclusion criteria were collated into a data extraction table (see Supplementary File 2). A double-blind selection procedure was performed by the first author and an HAS medical advisor. Instances of disagreement regarding the training tools’ selection were discussed together to reach a consensus. The selected training tools were then shared with the working group.

Semi-structured interviews with selected HTAbs and patient groups

The bodies that produced the selected training tools were contacted and semi-structured interviews were proposed, focusing on their experience in training patients in HTA. Eight interviews with representatives of HTAbs, one with a network of HTAbs, one with a research body involved in HTA, and two interviews with patient groups (see Supplementary File 1) were conducted by the first and last authors between January and April 2022.

Creation of a dedicated patient and consumer working group

Following a public call, between December 2021 and January 2022, thirty applications were received, and thirteen participants were selected. The selection criteria included members of small and large PCGs ensuring balanced representation of diseases (i.e., rare, and non-rare), gender, previous or non-participation in HTA at HAS. The study results were presented during the first working group meeting in February 2022 to help target the needs and preferences of the group.

Results

The results are reported in seven parts: (i) articles found through the literature search in French and international databases, (ii) training tools identified and selected from French and international websites as indicated in the methods section, (iii) key themes addressed from the analysis, (iv) formats identified, (v) identified accessibility methods (including cost), (vi) identified assessment methods, and finally, (vii) the analysis/exploration of themes and choices of the working group.

Articles found through the literature search in French and international databases

Searches performed in the French database and the international literature search on Medline and Cochrane Library databases did not result in the identification of any relevant article.

Training tools identified and selected from French and international websites

Eighty-two online training tools were selected according to the specified inclusion criteria The details of the online training tools selected other than those proposed by the HAS are found in Supplementary File 2.

HAS

Eight online tools related to HTA at HAS were developed before November 2021:

• • Four methodological guides for both patients and consumers on the HAS website (early access to medicines, medicines’ reimbursement and medical device reimbursement procedures, and patient groups’ participation)

• • One recorded symposium in 2016 covering several topics related to the patient perspective in HTA, featuring international guest speakers,

• • PowerPoint presentation notes for the information day on medical device assessment in 2018 aimed at patient groups,

• • Two webinars in 2020 and 2021, both on HAS’ YouTube channel, were organized to reinforce PPI in the work of HAS, covering a broad range of patient and consumer topics in addition to HTA.

At the time of this research, no specific training had been developed to educate PCGs with the objective of their subsequent contributions to the HTA process at HAS.

Bodies other than HAS

International bodies involved in HTA

We found that fourteen HTAbs and one network of HTAbs (RedETS) involved in HTA delivered training tools:

• • Agency for Care Effectiveness (ACE – Singapore),

• • Belgian Health Care Knowledge Centre (KCE – Belgium),

• • Canadian Agency for Drugs and Technologies in Health (CADTH – Canada),

• • Center for Drug Evaluation (CDE – Taiwan),

• • Gemeinsamer Bundesausschuss (G-BA – Germany),

• • Health Information and Quality Authority (HIQA, Ireland),

• • Health Intervention and Technology Assessment Program, Ministry of Health (HITAP – Thailand),

• • Health Technology Wales (HTW – Wales),

• • HTA Consumer Evidence and Engagement Unit, Department of Health (Australia),

• • National Institute of Care Excellence (NICE – England),

• • Institute for Quality and Efficiency in Health Care (IQWIG – Germany),

• • Institut national d’excellence en santé et en services sociaux (INESSS – Québec, Canada),

• • Scottish Health Technologies Group (SHTG – Scotland),

• • Scottish Medicines Consortium, Healthcare Improvement Scotland (SMC – Scotland),

• • Red Española de Agencias de Evaluación de Tecnologías Sanitarias y Prestaciones del Sistema Nacional de Salud (RedETS – Spain).

European and international PCGs

We identified nine European and international PCGs with training tools:

• • Beacon UK,

• • European Cancer Patient Coalition (ECPC),

• • European Organization for Rare Diseases (EURORDIS),

• • EUPATI,

• • International Alliance of Patient Organizations (IAPO),

• • Irish Platform for Patient Organizations, Science, and Industry (IPPOSI),

• • Lupus Europe (LE),

• • Myeloma Patients Europe (MPE),

• • Patient Voice Initiative (PVI, Australia).

Other organizations

The following thirteen organizations have training tools:

• • Agence Nationale de sécurité du médicament et des produits de santé (ANSM),

• • Agence Nationale Sécurité Sanitaire Alimentaire Nationale (ANSES),

• • European Medicines Agency (EMA),

• • EUnetHTA,

• • Guidelines International Network (GIN),

• • HTAi,

• • INAHTA,

• • International Society for Pharmacoeconomics and Outcomes Research (ISPOR),

• • NIHR England,

• • National Library of Medicine: National Information Center on Health Services Research and Health Care Technology (NLM-NICHSR – USA).

• • National Health Council, (NHC- USA),

• • Patients Active in Research and Dialogues for an Improved Generation of Medicines (PARADIGM),

• • World Health Organization (WHO).

Key themes addressed from the analysis

Upon identification of these training tools across the 38 French, European, and international bodies identified above, an analytical approach was taken to categorize the content involving screening for common themes (any theme recurring across three or more bodies). The themes were then classified, with input from the working group, into two categories: those related to HTA, and those related to PPI in HTA.

Key themes relating to HTA

Five key themes were found within the training tools (Table 1):

1. 1. What is HTA?

2. 2. What is the HTA procedure of a medicine, medical device, or professional practice?

3. 3. Definition and explanation of clinical-effectiveness assessments and/or economic assessments

4. 4. Methods of information collection for HTA and the type of information required to conduct HTA?

5. 5. Definition and explanation of the outcomes of HTA

Table 1. Number of selected bodies delivering one or more training tool(s) containing key themes relating to HTA

The two most common themes were “Methods of information collection for HTA and the type of information required to conduct HTA?” and “What is HTA?” (30/38 or 79 percent)The training tools identified often used visual illustrations to demonstrate the HTA procedures (notably the assessment, appraisal, decision-making HTA stages) of a medicine, medical device, and professional practice (i.e., flowcharts and timelines for a pharmaceutical HTA for reimbursement purposes). Definitions and explanations of clinical-effectiveness assessments and/or economic assessments were also identified within the selected training tools. Within this theme, elements of clinical-effectiveness included defined population and subgroups, level of innovation, comparison with existing treatments, etc., whilst others included economic elements (i.e., price, cost-effectiveness, health-related quality of life impact, etc.). Methods of information collection required for HTA included an explanation of systematic reviews (i.e., synthesis of all available research for a particular topic, for example, a summary of all available clinical trials) for most of the training tools selected. The information in most training tools contained the usual type of data required to conduct HTA, that is, quantitative and qualitative studies. The definition and explanation of the outcomes of HTA varied: for example, in terms of clinical utility, the medical service provided, or improvement of the medical service provided for a pharmaceutical clinical-effectiveness assessment at HAS, or an economic assessment including the incremental cost-effectiveness ratio (ICER) and quality-adjusted life years (QALY) at NICE and HIQA.

Key themes relating to PPI in HTA

Seven key themes relating to PPI in HTA were found (Table 2):

1. 1. How does a patient and/or consumer contribute to HTA?

2. 2. How are patient and consumer views considered by the HTAb for technology assessment?

3. 3. The HTAb’s governance and the role of the PPI department within body,

4. 4. What can HTAbs expect from an HTA patient/consumer contribution?

5. 5. Confidentiality and ethical practices, standards, and/or values,

6. 6. References and other resources (e.g., glossary),

7. 7. Soft skills (e.g., communication).

Table 2. Number of selected bodies delivering one or more training tool(s) containing key themes relating to PPI in HTA

The most common theme related to PPI in HTA identified was “How does a patient and/or consumer contribute to HTA?” (30/38 or 79 percent), followed closely by “How are patient and consumer views considered by the HTA body for technology assessment?” and “What can HTA bodies expect from an HTA patient/consumer contribution?” (28/38 or 74 percent). Most HTAbs accepted patient and consumer submissions via questionnaire; some held patient hearings during appraisal, and this was communicated together with how the input would be integrated into the final HTA report. Tools provided by HTAbs (HTW, REDETS, SHTG, SMC) included examples of quantitative and qualitative information. The level of detail on patient and consumer submissions and their inclusion varied, some provided step-by-step methods in guide format depending on the type of participation (HAS, KCE, NICE, REDETS), whilst others provided a brief overview on a webpage or slides (HTA-AU, SMC). Furthermore, ACE, EUPATI, and HTAi had their own glossary of HTA-related terms. Training on soft skills (e.g., communication methods for videoconference participation, in-person patient hearings) for optimal patient and consumer participation was highlighted through the semi-structured interviews by G-BA and NICE who train on these aspects. CADTH, NICE, and HTW included all seven topics in their training materials.

Formats identified

The formats of each tool were examined and classified as interactive and non-interactive. The definition of an interactive training tool is one where patients and consumers can interact with or provide feedback to the trainers in real time. This occurs usually in the form of a question-and-answer (Q&A) period, via an online forum, or exit feedback survey sent following each internal module for example.

Interactive formats identified

Four interactive formats were found (Table 3):

1. 1. Webinar (with live Q&A),

2. 2. Online symposium (with live online forum and live Q&A),

3. 3. Internal online modules (with online forum and feedback survey),

4. 4. Massive Open Online Course (MOOC) (with online forum and feedback survey).

Table 3. Number of selected bodies delivering one or more training tool(s) through an interactive or non-interactive format

The most common interactive format was a webinar with Q&A (8/38 or 21 percent). Internal online modules covering several themes were only delivered by two organizations: NICE and ISPOR. A MOOC was developed by NIHR in England and three international patient groups: Beacon UK, EUPATI, and EURORDIS.

Non-interactive formats identified

Seven non-interactive formats were found (Table 3):

1. 1. Webpage(s),

2. 2. Patient and consumer guides in Word/PDF format,

3. 3. PowerPoint slides,

4. 4. Webinar (without Q&A),

5. 5. Online toolkit with text and videos,

6. 6. Short videos,

7. 7. E-book.

The two most common non-interactive formats used by the identified bodies were: patient and consumer guides in Word/PDF format (13/38 or 34 percent) and webpage(s) (12/38 or 32 percent), followed by short videos (of 15 minutes or less each explaining a key theme identified) (8/38 or 21 percent). Online contents were displayed either on a single webpage (e.g., ME), or series of webpages (e.g., HAS, CADTH). Certain short videos were delivered via animation (e.g., HTAi, CADTH, CDE) whilst others had a speaker delivering the message (e.g., SMC, ACE).

Identified accessibility methods (including cost)

The training tools identified and selected by HAS were classified according to their accessibility methods including cost (Table 4):

• • Accessible without cost,

• • Accessible without cost following online subscription, after the creation of an account and password,

• • Available for a fee only via online subscription, via creation of an account and password,

• • Available online 24 hours a day, 7 days a week,

• • Accessible for a limited period on demand with or without cost.

Table 4. Number of selected bodies delivering one or more training tool(s) according to their identified accessibility methods

Most of the tools were accessible freely without subscription (33/38 or 87 percent), 24 hours per day, 7 days a week (32/38 or 84 percent).

Identified assessment methods

Assessment tools (Table 5) to test the patient or consumer’s understanding of the HTA-related information communicated were provided by 6/38 or 16 percent of the selected websites. Amongst those with an assessment method, a quiz with multiple choice answers or a series of short answer questions was the most common method employed, for example, following a MOOC (e.g., EUPATI, EURORDIS, NIHR).

Table 5. Number of selected bodies delivering one or more training tool(s) according to their identified assessment methods

Analysis/exploration of themes and choice of the working group

Upon presentation of the study, the objective of the first working group meeting was to define needs and preferences for the future HAS training tools in terms of themes, format, accessibility, and assessment. Two key ideas were prioritized:

1. 1. Construction of a knowledge-base via PowerPoint in two blocks of modules (key themes related to HTA and to PPI in HTA) containing essential information for PCGs for HTA participation, targeting PCGs that have never contributed to HTA at HAS. This tool should be available 24/7 and contain an assessment quiz.

2. 2. Development of a short video (3 to 4 minutes) for patients and consumers to explain “Why it is necessary for patients and consumers to participate in HTA”

Discussion

The study results were analyzed to identify similarities and differences between the themes and format delivered by the different bodies.

The three most common themes identified in the research were What is HTA?, Methods of information collection for HTA and the type of information required to conduct HTA? and How does a patient and/or consumer contribute to HTA? Each of which is known to be important in PPI in HTA (14–16). Besides these three themes, the nine others detailed in the results were selected as they were recurrent topics in the specific training modules (internal modules or MOOCs) delivered across the European and international PCGs and certain HTAbs with comprehensive training tools (HIQA, NICE, REDETS).

Despite their importance, the themes of confidentiality, ethical practices, standards, and/or values (17; 18) in HTA were addressed by just over half of HTAbs and two-thirds of European and international PCGs. This was unexpected given that HTAi has published a series of values and standards for patient involvement in HTA (19; 20). Soft skills training such as communication methods were only discussed amongst 8/38 or 21 percent of identified bodies despite the COVID-19 pandemic highlighting the need for innovative PPI, including patient engagement in virtual meetings (21; 22).

The HAS PCG working group determined their preferences and needs for training: Individual review and selection of tools and themes proposed followed by a second collective selection of themes by the group. The collective results almost mirrored that of the study with the main two choices being: What is HTA? and How does a patient and/or consumer contribute to HTA? The working group appreciated tools that were able to deliver these topics with terms that were easy to understand and including schemas to explain complex processes.

After the group consolidated their responses, it was suggested that the knowledge-base should be produced in two sets of modules containing the following themes:

1. 1. Understanding HTA and the role of HAS

   1. a. HAS and its governance

   2. b. What is HTA?

   3. c. HTA of a medicine, medical device, professional practice

2. 2. PPI in HTA at HAS

   1. a. Patient and consumer participation in HTA at HAS: a practical guide

   2. b. HAS’ expectations and advice on patient and consumer data collection methods

A final module was added to the first set of modules later in the development process focusing on essential information for patients and consumers concerning changes due to the new European regulation (23).

Although less than a quarter of the selected training tools contained a glossary, the group suggested including one to explain key definitions. Those developed by HTAi and EUPATI were presented to the working group after translation into French, and it was decided that the EUPATI glossary was more accessible for persons unfamiliar with technical terms and with limited health literacy.

Consensus was reached that this first training tool would be produced as a PowerPoint slide deck. The working group appreciated this format as it would be easily accessible, and could be modified whenever relevant (i.e., for training purposes, feedback, etc.). The group guided the choice for two levels of information presented in the slide deck: primary information to understand the topic displayed on the slides accompanied by explanatory notes giving more detail and references. The tools identified only delivered one level of information. The working group felt that two levels of information were necessary to accommodate the needs of PCGs who are new to HTA and PPI in HTA at HAS and those of PCGs already accustomed to contributing to HTA at HAS. The slide deck is to be accompanied by a self-evaluation quiz. This format would enable HAS and PCGs to organize training webinars in the future. The training tool will be reviewed annually by the interface committee of the “Strengthening public involvement in HTA at HAS" roadmap for necessary updates and modifications. The group also desired to develop a short video highlighting the necessity and practicalities of PPI at HAS, as a communication tool encouraging PCG participation in HTA. These two training tools would be free and available 24 hours a day, 7 days a week on HAS’ website. They were reviewed by the SEU and found to be feasible within the budget and project schedule.

Limitations of the study

The study was not a comprehensive overview of available training tools. There are most likely other training tools that exist in a specific disease area. However, the research achieved its main objective: it provided valuable information to trigger the HAS working group’s reflection on the development of a French-specific training tool. The first aim of the study was to identify and analyze available training tools according to their themes, formats, and modalities of accessibility and assessment. Criteria to assess the quality of the resources found through the research were not defined, while it is indeed important in scoping studies (24). We noticed that the depth of information delivered by the training tools varied. Another limitation was the exclusion of languages other than English or French: some HTAb have specific guidance in their national languages, for example, Comissão Nacional de Incorporação de Tecnologias no Sistema Único de Saúde (CONITEC) in Brazil (6). The members of the working group judged which training tools they preferred according to their individual needs. Although the group members were chosen to ensure the best possible representation of French patients and consumers, their preferences may not have been holistically representative of the preferences of all French patients and consumers. Finally, the inclusion criteria for the training tools were limited (e.g., the source of the training tools to be included was limited to HTAbs and international PCGs and certain relevant other organizations). Smaller local PCGs, pharmaceutical companies, and universities that may have developed similar and useful training tools did not come up in the search results.

Conclusion

This international study identified and selected HTA training tools and materials for patients and consumers that can be grouped into two broad categories: HTA and PPI in HTA.

Considering the variability of formats, level of information presented in the tools, and the specific needs and preferences of patient and consumers, HAS decided to develop its own training tools for PCGs to improve their HTA participation in the future. A working group composed of patient and consumer experts guided the choices for the themes and formats, resulting in two tools with different purposes. Firstly, a knowledge-base slide deck with two levels of information: primary information displayed on the slides and accompanying notes and references allowing the reader to delve deeper into the subject, and a second, complementary tool - a short video explaining why patients and consumers’ participation in HTA is necessary, encouraging their participation in HTA at HAS. Both were published, in French, on the HAS website in November 2022(25).