Real-world evidence: perspectives on challenges, value, and alignment of regulatory and national health technology assessment data collection requirements

Abstract

Objective

This study aims to assess stakeholder perceptions on the challenges and value of real-world evidence (RWE) post approval, the differences in regulatory and health technology assessment (HTA) real-world data (RWD) collection requirements under the German regulation for more safety in drug supply (GSAV), and future alignment opportunities to create a complementary framework for postapproval RWE requirements.

Methods

Eleven semistructured interviews were conducted purposively with pharmaceutical industry experts, regulatory authorities, health technology assessment bodies (HTAbs), and academia. The interview questions focused on the role of RWE post approval, the added value and challenges of RWE, the most important requirements for RWD collection, experience with registries as a source of RWD, perceptions on the GSAV law, RWE requirements in other countries, and the differences between regulatory and HTA requirements and alignment opportunities. The interviews were recorded, transcribed, and translated for coding in Nvivo to summarize the findings.

Results

All experts agree that RWE could close evidence gaps by showing the actual value of medicines in patients under real-world conditions. However, experts acknowledged certain challenges such as: (i) heterogeneous perspectives and differences in outcome measures for RWE generation and (ii) missing practical experience with RWD collected through mandatory registries within the German benefit assessment due to an unclear implementation of the GSAV.

Conclusions

This study revealed that all stakeholder groups recognize the added value of RWE but experience conflicting demands for RWD collection. Harmonizing requirements can be achieved through common postlicensing evidence generation (PLEG) plans and joint scientific advice to address uncertainties regarding evidence needs and to optimize drug development.

Background

The pharmaceutical industry, regulators, and health technology assessment bodies (HTAbs) are increasingly exploring the potential of real-world data (RWD) as a complementary source to generate evidence regarding the effectiveness and safety of medicines in routine clinical practice (1;2). RWD is defined as an overarching term for data (such as efficacy or safety) collected outside of the context of conventional randomized controlled trials (RCTs) (2). The generated evidence obtained through the synthesis and analysis of these raw RWD is defined as real-world evidence (RWE) and may affect healthcare decision making by filling evidence gaps for regulatory decisions along a medicine's lifecycle (2;3). Indeed, the European Medicines Agency (EMA) aims to increase the use of RWE in regulatory decisions and has been relying on RWD collected on an “ad-hoc” basis for postmarket safety surveillance and noninterventional postapproval safety and efficacy studies (PASS, PAES) to generate additional evidence on drug utilization and adverse outcomes in practice (4–7).

Furthermore, RWD collection requirements can become mandatory in Germany following the new regulation for more safety in drug supply (GSAV) (8). Under this regulation, the Federal Joint Committee (G-BA) may now request RWD collection from pharmaceutical companies to be submitted to compulsory indication-based registries for the national benefit assessment of new medicines that can only show “limited scientific evidence” such as orphan drugs, drugs conditionally approved, or those approved under exceptional circumstances (9–11). The exact requirements regarding the duration, nature, scope, and methodology of RWD collection according to patient-relevant end points still need to be determined before RWD collection can potentially be requested. Despite the increased attention on the potential value of RWE for regulatory and reimbursement decisions, the collection and the use of RWE are burdened with operational, technical, and methodological challenges (12–14). Additionally, the GSAV may lead to double structures at the regulatory and the national levels with heterogeneous evidence requirements that do not contribute to an improved RWD collection (15). Hence, there is a need to align requirements to enable optimal evidence appraisal and to maximize the utility of RWE throughout drug development. However, there is still a knowledge gap regarding stakeholder perspectives on the sequencing of regulatory and HTA postevidence requirements, that is, how to create a complementary system that employs RWE in synergetic postapproval studies. Accordingly, this study aims to assess stakeholder perceptions regarding the different requirements for postapproval data collection at the regulatory level compared with the requirements for national HTA follow-up in Germany. The purpose was to gain an in-depth knowledge about stakeholder views regarding the challenges and added value of RWE as part of postlicensing evidence generation (PLEG) and the obstacles of the two separate systems and to determine how experts envision future solutions to align national HTA with the EMA regulatory scheme to create a complementary framework for RWE requirements.

Methods

Semistructured interviews were conducted with experts in the field of RWE. An interview guide was developed based on a document analysis of reports on EMA and German RWE requirements and their analysis for regulatory purposes and HTA (5;16;17). The final questionnaire consisted of seven open-ended questions and was divided into two main parts: first asking for the perceived value and challenges of RWE post approval and then asking for the experienced differences in requirements and potential opportunities for alignment. The questions were slightly adjusted in four of the interviews with experts outside of the German healthcare context to obtain their experience with national evidence requirements in their country instead of their views on the GSAV. Content validation was done by two additional researchers; the questionnaire was adjusted accordingly to create a clear and informative questionnaire. The final version of the questionnaire can be found in Table 1. For German respondents, the questionnaire was adjusted and directly translated into German by a native speaker of the research team.

Table 1. Interview questions

Note: RWE, Real-world Evidence; HTA, Health Technology Assessment.

A total of eleven semistructured interviews were conducted with RWE experts from the pharmaceutical industry (4), regulatory agencies (2), HTAbs (3), and academia (2). Interviewees from the pharmaceutical industry included experts from two different companies, the European Federation of Pharmaceutical Industries and Associations (EFPIA) and the German Association of Research-Based Pharmaceutical Companies (VFA). The regulatory group included two experts from the EMA, whereas the HTAb group included experts from the G-BA, the Institute for Quality and Efficiency in Healthcare (IQWiG), and the Swedish TLV. For the academic group, a health economist from a German consulting agency and an expert from the private institute for applied health service research (inav GmbH) were interviewed. The interviewees were purposively selected and recruited through personal contact using e-mail. Interviewees were mainly from Germany, but other experts from England (ENG), Sweden, the Netherlands, and Belgium were also included to understand their experiences with national RWE requirements in other European countries. After the interviewees agreed to participate, teleconferences were arranged. The interview was recorded upon given consent, transcribed afterward, and translated to English if needed. Additionally, a summary of the transcript was sent back to the interview participants for a member check to ensure the credibility of the collected data. Once the interview data had been checked, the analysis was performed by using the NVivo software and the recommended coding approach in the literature (18). Ethical approval was obtained from the ethics assessors of Maastricht University and the research was classified as a low-risk project (FHML/HPIM/2020.087). The participants were approached in their professional function only and did not necessarily represent the view of their institutions. They agreed to participate voluntarily. Confidentiality of the interview data and personal anonymity of the interviewees were assured.

Results

The results are classified according to the topics of the interview questionnaire. An overview of the main findings according to stakeholder group (P, R, HTAbs, or A) can be found in Table 2. The numbers behind each statement indicate the number of interviewees who mentioned a specific opinion.

Table 2. Summary of the main interview findings

^a Number of interviewees who described a specific issue.

Note: P, Pharmaceutical industry; R, Regulators; HTAbs, HTA bodies; A, Academia; RWE, Real-world Evidence; RCTs, Randomized Controlled Trials; HTA, Health Technology Assessment; EMA, European Medicines Agency; AMNOG, Act on the Reform of the Market for Medicinal Products; AIFA, Italian Medicines Agency; G-BA, Federal Joint Committee.

Added Value and Challenges of RWE

Current Role of RWE Post Approval

The role of RWE post approval differs between the regulatory and the HTA level. Pharmaceutical and regulatory experts agreed that RWE is commonly a part of regulatory requirements for PASS and PAES. RWE typically supports postapproval safety evaluations by monitoring spontaneous adverse events or is part of the requirements for long-term documentation of the safety and efficacy of Advanced Therapy Medicinal Products (ATMPs) (19). Additionally, RWD collection is often linked to managed entry agreements and conditional marketing approvals. In comparison, five respondents (P, HTAbs, A) agreed that RWE has virtually no scientific role for HTA. Most HTAbs still prefer evidence from RCTs as the gold standard for the national benefit assessment. Consequently, four respondents (P, HTAbs, A) agreed that RWE is not the appropriate manner for the German HTA process. RWE is rarely submitted by pharmaceutical companies as RWD does not exist at the time of benefit assessment after marketing approval.

Perceived Challenges and Value

Regarding the challenges of RWE, six respondents from all stakeholder groups mentioned methodological problems with RWE such as missing randomization or selection bias. Similarly, the quality of RWD was perceived as a major challenge by five respondents (P, R, HTAbs) as often no quality control infrastructures are in place to ensure the completeness of data collection. Furthermore, a lack of standardization in RWD collection leads to differences between countries, regions, and hospitals. Two respondents (P) further mentioned the challenge of data acceptability. Respondents often perceived reluctancy and philosophical objections in the healthcare sector, leading to a lack of trust by HTAbs to use RWE in decision making. Despite these challenges, all respondents saw the potential value in the use of RWD as a source to complement clinical data for evidence generation. Four respondents (P, HTAbs, A) mentioned that RWE shows the actual value of medicines in real patients with different lifestyles and comorbidities. Experts from the pharmaceutical industry and the HTAbs valued that RWD collection does not take place with strict study protocols, allowing the inclusion of patient populations with worse general health conditions. According to four respondents (P, HTAbs), RWE may also reduce uncertainty by filling knowledge gaps that could not be addressed in RCTs. Hence, two respondents (P, R) mentioned that RWE could step in when RCTs are not ethically or operationally feasible.

Most Important Requirements

Two experts (R, HTAbs) explained that the evidence requirements generally depend on the specific research question of interest that determines the appropriate data sources and quality characteristics. Moreover, most respondents (6) mentioned the high quality of RWD as the most important requirement. Two respondents (P, R) perceived a good understanding of the data origin and the completeness of well-controlled and systematically collected data as important.

Additionally, four stakeholders (HTAbs, A) agreed that a comparator is the most important basic methodological requirement for real-world studies. Respondents stressed that comparative assessments must be possible in real-world studies to generate meaningful comparative information for the benefit assessment. The HTA experts further stressed the importance of defining patient-relevant end points for RWD collection, preferably in indication-based registries.

Experience with Registries as a Source for RWD

Registries were an often-mentioned source of RWD in the postapproval context. However, three respondents (P, HTAbs, A) agreed that there are few high-quality registries in Germany that could be used to collect RWD. Additionally, the German HTAb experts mentioned a limited registry practice at the national level. The German HTAbs want to see certain endpoints such as quality of life (QoL) that are not adequately collected in registries. Hence, existing German registries may not be suitable for use to generate additional evidence related to many indications. At the regulatory level, there is experience with registries as a data source for authorization. However, there is also a diverse and heterogeneous landscape of registries in Europe, including professionally setup registries, that meet important data transparency and management requirements, but also smaller registries with lower quality due to financing problems. Regarding this, three respondents (R, A) mentioned the EMA registry initiative that was set up to improve the quality of registries as a source for RWD collection in Europe. According to two respondents (HTAbs, A), different research questions from regulators and HTA could be answered using the same registry as a data source if an indication-based registry for ongoing data collection is in place.

Differences in Requirements and Alignment Opportunities

Perceptions Regarding the GSAV Regulation

The GSAV is perceived as a controversial law leading to uncertainty regarding the implementation rules (P, R, A). Most respondents believed that the new law might not be able to meet the high expectations in practice due to the methodological challenges. According to two respondents (P, HTAbs), the law intends to create better alignment between the EMA and the AMNOG system during the national benefit assessment and to ensure completeness of data collection with appropriate commitments. The German HTAbs perceive the GSAV as a useful tool to limit the prescription authorization for new products to those service providers who participate in such RWD collection, whereas respondents from the pharmaceutical industry experience obstacles due to a lack of predictable planning. Stakeholders often assumed that such RWD collection would be mandatory in addition to existing regulatory postapproval data collection requirements. It remains unclear how the G-BA will identify compounds to request additional data and how methodological criteria are defined.

Moreover, four respondents (P, HTAbs, A) expect that additional RWD collection will not be a regular requirement for HTA. The experts agreed that the G-BA will consider the possibility of RWD collection carefully and will request it only in a few specific cases, most likely targeting medical products such as ATMPs, gene therapies, or oncological products. Potential cases may be products approved under exceptional circumstances (20), where there is a need for additional data, because no further evidence from RCTs can be expected. Two respondents (P, HTAbs) mentioned that a concept for the potentially mandatory RWD collection will be developed to define key points for study protocols. The concept will be the cornerstone for a study plan that pharmaceutical companies need to adhere to. However, the exact methodological requirements will be determined individually per case in a joint consultation at the national level. Nevertheless, the G-BA will need to take into account what the EMA is already demanding to ensure that the requirements are compatible.

Experience with RWE Requirements in European Countries

The additional national postevidence requirements following the GSAV appear to be a special case in Germany. From the interviews with non-German experts, it appeared that no similar mandatory law exists in other European countries. However, the National Institute for Health and Care Excellence (NICE) in ENG may, in certain cases, approve a drug only with further research and additional evidence. Similarly, in Sweden, recommendations are issued for pharmaceutical companies to validate their study results with registry data. The Netherlands also does not have a specific law, but it seems to have similar concerns regarding the lack of evidence as Germany and requires specific information if a drug is very expensive. Additionally, two respondents from the pharmaceutical industry recognized clear country differences regarding the openness of using RWE for healthcare decisions. For instance, two experts perceived Germany as lagging behind regarding the acceptance of RWE by showing more reluctancy toward evidence from real-world practice. In comparison, three respondents (P, A) mentioned ENG as a country with registry experience and interest of its HTAbs in RWD collection due to its experience with managed entry agreements. Also, Sweden has good access to many big and specified registries for various diseases, allowing the Swedish HTAbs to obtain RWE on medicines and patients. The other countries mentioned by respondents with a more extensive registry and RWE experience are Italy and Belgium.

Regulatory versus HTA Needs

Most respondents (8) from all stakeholder groups agreed that the regulatory focus for RWD collection is on confirming the safety profile of a product post approval. Four respondents (P, R, A) explained that for regulatory approval, the developer needs to show a positive benefit–risk for the specific product and, in most cases, noninferiority compared with the current standard of care or an alternate product. In comparison, five respondents from all groups agreed that HTAbs need to evaluate the added benefit compared with standard therapy. According to the respondents, HTAbs require RWE to show the relative effectiveness of a drug on patients in real-world conditions, confirming the efficacy profile established under controlled environments of RCTs in practice. Five respondents (P, R, HTAbs) mentioned that HTAbs need to show the cost-effectiveness of a medicine on a long-term basis, stressing the reimbursement aspect according to the priorities of a given healthcare system.

When comparing the different needs, seven stakeholders (P, R, HTAbs) perceived the main differences in purpose, questions, and perspectives guiding the two systems in their evaluation. The perspective of the EMA is strongly focused on the individual product, whereas the HTAbs are more interested in a larger overall picture to determine how a product performs compared with standard therapy. Hence, three respondents agreed that the different legal roles of the EMA and HTA in the system explain the need to have heterogeneous evidence requirements. Furthermore, there is a difference in outcomes. Six respondents from all stakeholder groups agreed that the study and data leading to authorization based on the safety and efficacy requirements do not always cover all endpoints that are required for HTA, such as patient-reported outcomes (PROs) or QoL. Additionally, three respondents (P) thought that the EMA is more open toward novel ways of generating data and more willing to accept uncertainty in the submitted evidence. On the contrary, the respondents perceived HTAbs to have a stricter view on evidence trying to avoid uncertainty due to the greater responsibility for care policy, costs, and interests of general society.

Opportunities for Alignment

Different opinions were raised regarding potential future alignment. Five respondents (P, HTAbs) agreed that stakeholders should coordinate more closely in light of the different objectives to create a continuous flow of evidence generation, align closer on specific criteria, and increase the acceptability of RWE. Furthermore, three respondents (P, HTAbs) saw the need to align comparators and outcome measures in comparative studies for both systems. Stakeholders hoped that in the future, HTA-relevant criteria may be requested in regulatory studies. The available Joint Scientific Advice platform that was mentioned by the majority of respondents (8) from all stakeholder groups is seen as a facilitation tool to bridge the gap between regulatory and HTA needs, discuss data governance, and lay foundations for coordinated evidence requirements.

However, respondents also mentioned that the actual success of the advice varies depending on the commitment of all parties to reach an agreement. Regarding this, five respondents (P, HTAbs, A) perceived the EUnetHTA as a useful platform to align evidence needs between national HTAbs. Again, concerns were raised regarding the future uncertainty of this offering and the degree of actual authority in the absence of a legal framework at the EU level. Two respondents (P, HTAbs) mentioned that the best time for a joint discussion would be as early as possible, but at least when the EMA is considering requirements for postapproval data collection during the approval process. At that time, the pharmaceutical companies may consider approaching the HTA with their proposal to see whether it would also cover relevant aspects from the HTA perspective. On the contrary, five respondents (R, HTAbs, A) did not see the possibility of complete alignment and mentioned that it will be inevitable to have separate requirements due to the different perspectives and legal basis.

Discussion

This study contributes to the discussion about the alignment and integration of regulatory postapproval and HTA evidence requirements. Different stakeholder groups perceive the main differences in RWE requirements to originate in the inherently different questions, timing, and purposes guiding RWE generation in the two systems. The difference between the regulatory approach to generalize and the HTA approach to contextualize the value of a drug in national healthcare settings is further challenged by the heterogeneity of RWD collection and its quality. All stakeholder groups experience methodological problems and high demands regarding the quality control infrastructure for RWD collection. However, stakeholders agree that RWD are promising for evidence generation and could complement the evidence package generated through RCTs. The new GSAV law in Germany is expected to improve the completeness of data collection, but the results demonstrate that practical experience with RWD from registries within the AMNOG process is missing as the implementation of the law is not yet clearly defined. The extensive registry experience from other European countries may help design the RWD collection in Germany. Moreover, complete harmonization of evidence requirements may not be possible due to the different legal frameworks, but aligning the data collection for evidence generation should be possible. Engaging in early discussions with regulators and HTAbs could lead to postapproval studies that are aligned on comparators and outcomes to meet the requirements of both systems.

Previous studies have looked at differences in regulatory and HTA evidence requirements. Similar to observations made in this study, an analysis of key expert opinions showed that the value of RWE was recognized by all stakeholders, but regulatory decisions tend to focus on RWE for efficacy and safety while HTA is assessing cost-effectiveness (21). Moreover, the willingness to accept RWE for decision making varies between regulatory authorities and HTAbs. The EMA has experience with RWD collection used to monitor the long-term safety of drugs in practice and to support lifecycle benefit–risk evaluations (13;22). In the German HTA context, stakeholders still perceive hurdles with the usage of RWE in terms of quality and privacy of RWD (21). The interview findings support this observation. Industry experts experience regulators to be more open toward novel ways of generating RWE and show more willingness to accept uncertainty compared with the German HTAbs who seem to have a more scalar view on the risk of uncertainty of evidence. Nevertheless, the stakeholder views in this study suggest that having uniform standards for data collection and aligned PLEG plans may help work toward the goal of generating RWE that is accepted in both contexts. Pharmaceutical companies should be actively engaged in discussions and come forward with proposals early on to determine which type of real-world studies or data sources could be suitable for the required PLEG plan. This is an observation also made in a recently published review on PLEG advice (22), suggesting that common PLEG programs could help improve market and patient access due to the streamlined evidence development.

This research has demonstrated that the pharmaceutical industry is trying to satisfy both evidence requirements, but the approval studies rarely cover HTA needs due to the use of different comparators and outcome measures as defined by regulatory guidance. The German case demonstrates that future national RWE requirements should be relying on and complementing already mandated RWD collection by the EMA to integrate additional HTA end points into ongoing regulatory postapproval studies in indication-based registries. Furthermore, to recognize the value of joint work, this study suggested that it is imperative to have a permanent working structure that can consistently engage all stakeholders, such as EUnetHTA in partnership with the EMA. It seems that the German HTAbs may not necessarily participate in scientific advice procedures due to the absence of a European legal framework. However, such platforms can increase mutual understanding regarding methodological approaches and can support the establishment of operational arrangements (23). This study has several strengths. To our knowledge, this study is the first international attempt of using semistructured interviews to outline stakeholder positions in the discussion about the possible use of RWE in drug assessment. Due to the qualitative nature of the study, unique insights into the individual perspectives of key experts from the pharmaceutical industry, regulatory and HTA agencies, and academia were captured. The results of this research could also serve as a starting point for a more extensive study including a larger number of interviewees from the different stakeholder groups and experts from payer associations. Future research using case examples for which national RWD collection obligations were introduced in Germany would add to the discussion on aligned evidence requirements for drug development in practice. Assessing the actual consequences of heterogeneous RWE requirements will only be possible in the future once the G-BA decides on the first cases for which the additional data collection applies.

There are potential limitations to this study. First, only one researcher did the data analysis and coding due to time restrictions. More versatile results could have been achieved by performing the data analysis with two researchers. Additionally, the research was performed while the exact German RWE requirements were not yet published by the G-BA. The research was, therefore, only able to capture the expectations of stakeholders regarding the differences in evidence requirements. It would be interesting to receive further insights into the actual experiences of different stakeholder groups when RWD collection requirements will be in place. Also, even though data saturation was reached in some groups (P) and key experts were selected, the research was conducted with a limited number of participants, who responded in their personal capacity. Particularly, few regulatory experts and HTAbs were included. Another limitation may be that the study was conducted mainly with German experts. Other countries could have different requirements.

Conclusion

Concluding, this study suggests that different stakeholders experience conflicting demands for RWE generation. Regulators require RWE to complement the understanding of a medicinal product’s safety to confirm its benefit–risk profile, whereas for HTAbs, RWE should allow for comparative assessments and close evidence gaps to establish the added value of a new product. In the German case, the exact requirements are highly anticipated and will determine the scope and methodology required for the national benefit assessment. Stakeholder perspectives indicate that RWD collection will take place only on a rare, case-to-case basis. The next step for all stakeholder groups should be to fully engage in PLEG scientific advice regarding RWE requirements early on to anticipate evidence needs and optimize drug development to ensure quick patient access. Early collaboration can help bridge the gaps between regulatory and HTA perspectives to reduce unnecessary cost and duplication of postapproval studies conducted by pharmaceutical companies, which may delay patient access.