A systematic literature review of real-world evidence (RWE) on post-market assessment of medical devices

Abstract

The increasing use of real-world evidence (RWE) and real-world data (RWD) to assess post-market medical devices (MDs) might satisfy the urgent need for data sharing and traceability. This study sought to (i) get an overview of current practice in post-market assessments of MDs reporting on RWE/RWD; (ii) draw policy recommendations for governments and health organisations and identify a research agenda for scholars.

A systematic review was undertaken until February 2024 following the PRISMA guidelines. Original peer-reviewed articles in English and incorporating RWE/RWD into any sort of post-market assessment strategy for an MD were included and their reference lists manually checked. A narrative synthesis was employed to describe evidence retrieved.

Totally, 145 research articles were identified. Administrative databases were mostly utilised; clinical and/or economic evidence gathered in a short/medium time horizon the most frequently reported; other evidence types (e.g., organisational) underreported; patient perspectives rarely incorporated; the innovation complexity of MDs relatively low.

To our knowledge, this study is the first in its kind to provide a comprehensive picture of how non-randomised evidence has been used when assessing MDs working in real-life conditions. The implications of this review might help health policy scholars in addressing the avenues for research in RWE for MDs and policy-makers to better understand the risks and benefits of medium and long-term use of MDs alongside clinical practice and make more informed decisions about adoption and use.

1. Introduction

The role of evidence in healthcare, and the way evidence is used to inform decisions on technology introduction and adoption, differs from other industries (Barlow, 2020). This is because new technologies are taken-up within a complex environment made of: intertwined policies and regulations at both institutional and organisational level, multiple professional cultures, and different stakeholders who take part in the decision-making process (Herzlinger, 2006; Fung et al., 2008; Roth, 2019; Barlow, 2020). The importance of evidence is partly related to how decisions on adoption and diffusion of technology innovations are made within the healthcare ecosystem, and partly to the methods used for assessing innovations, aspect intrinsically linked to the basis of that evidence (Haynes, 1990; Lomas, 2007; Barlow, 2020). Evidence generation, interpretation, and validity are complex and controversial, as each step needs to be judged both pertinent and sufficient from a variety of professional groups operating within the healthcare ecosystem, including representatives of a wide range of organisations and institutions (Davies et al., 2000; Webster and Wyatt 2020; Mackenzie et al., 2010; Savory and Fortune, 2013). In this complex system, each decision about the introduction and spread of an innovation needs to engage and persuade all active stakeholders on board. Nevertheless, (i) the techniques for evidence generation may be underdeveloped and positivist scientific methods, such as randomised controlled trial (RCTs), may be not appropriate when assessing complex innovations like medical devices (MDs); (ii) different stakeholders may have different expectations on what constitutes evidence and the evidence basis, as well as contest its interpretation; (iii) there may be no agreed criteria to assess evidence validity (Barlow, 2020). Health Technology Assessment (HTA) has been adopted worldwide as a cross-disciplinary and multidimensional measurement framework for evaluating the performance of a medical technology at different time points of its lifecycle (O'Rourke et al., 2020). In a broader term, HTA is the systematic evaluation of the clinical, health economic, societal, legal, and ethical issues related to the introduction, dissemination, and use of a medical technology (Banta, 2003). It aims to generate and synthesise multi-disciplinary evidence to inform health policy, resource allocation, and clinical decision-making (Manetti et al., 2020a, 2020b). An international joint task group co-led by the International Network of Agencies for Health Technology Assessment (INAHTA) and Health Technology Assessment International (HTAi) has developed a new and internationally accepted definition of HTA, that is: ‘HTA is a multidisciplinary process that uses explicit methods to determine the value of a health technology at different points in its lifecycle. The purpose is to inform decision-making in order to promote an equitable, efficient, and high-quality health system’(INHTA, 2024). The concept of HTA is intrinsically embedded with the approach of evidence-based to medicine and management, being an integral component of healthcare governance to set guidelines and standards, provide feedback and forwards actions on delivery of care, and improve quality and performance of health services alongside clinical practice (Madden, 2012; Barbazza and Tello, 2014; Nuti, 2016; Tarricone et al., 2017; Barlow, 2020). Historically, the object of HTA has been restricted to pharmaceuticals, rather than MDs, and evidence on clinical efficacy/effectiveness (i.e., can it work/does it work) and/or cost-effectiveness (i.e., is it worth it) has formed a key part of the formal assessment, taking over other relevant evidence types (e.g., human factor) when assessing the impact of a healthcare innovation (Schmitz et al., 2016; Barlow, 2020; Manetti et al., 2020a, 2020b). In terms of evidence generation, HTA has been traditionally based on positive scientific methods, such as systematic reviews (SR) and RCTs, which have been preferred due to their lower risk of bias by design compared to real-world studies (Kent et al., 2021). However, traditional methods for evidence generation, such as RCTs, raise general concerns about generalisability and external validity (Maggioni et al., 2016; Fuchs et al., 2017; Makady et al., 2017; Tarricone et al., 2017; Torbica et al., 2017). Moreover, such methods assume or imply that useful data on an innovation can be gathered according to study design, and this assumption is no longer appropriate when assessing complex healthcare innovations, like MDs, whose key features differ from other medical technologies and demand ‘a more pluralist approach to gather evidence on their impact’ (Tarricone et al., 2017; Barlow, 2020). In terms of efficacy/effectiveness, MDs are performance dependent on user skills and training, have a learning curve, may be used to treat different conditions in different clinical settings and present a faster product lifecycle (Ciani et al., 2017; Fuchs et al., 2017; Tarricone et al., 2017). In this sense, MDs represent a ‘dynamic’ innovation, whose attributes are not well-defined and specified, making trial results difficult to compare and quickly outdated (Crispi et al., 2019; Goring et al., 2019). This aspect has, in turn, a negative incentive on clinical evidence generation that is usually limited at each stage of an MD lifecycle and less stringent in terms of market approval than pharmaceuticals. Finally, MDs may bring together elements of new technology (i.e., physical innovation) and organisational process changes (i.e., service, staff, professional role) and, in this sense, are more complex to assess than traditional innovations (Crispi et al., 2019; Barlow, 2020). In recent years, there has been a growing interest in the use of non-randomised studies, which are becoming the main source of evidence for assessing MDs (Crispi et al., 2019; Kent et al., 2021). Real-world data (RWD) are data related to patient health and/or the delivery of routine clinical practice collected by multiple sources, such as registries, observational studies, health surveys, claims and administrative datasets, electronic health records (EHR), social media, mobile and wearable technologies to which MDs are connected (Garrison et al., 2007; Berger et al., 2017; U.S. Food and Drug Administration, 2017; Johnston et al., 2019). The related concept of real-world evidence (RWE), i.e., evidence obtained from the analysis of RWD, and the increased conduction of studies using RWE/RWD might satisfy the urgent need for data sharing, traceability, and help to understand the risks and benefits derived from medium and long-term use of MDs in routine clinical practice and current applications (Pane et al., 2019). Uncertainties and limitations concerning evidence on safety, efficacy/effectiveness, and cost-effectiveness, as well as rate of innovation uptake, are intrinsically linked to the special characteristics of MDs. Only limited and fragmented information is available on real-world performance of new or novel MDs, making challenging understanding what happens in real-life at different time points of the post-market phase (i.e., adoption, diffusion/monitoring, and obsolescence). Recent public scandals involving MDs after their successful introduction into routine clinical practice have raised medium-term safety concerns about public health showing the urgent need for evidence generation and monitoring (Pane et al., 2019). Indeed, postlaunch RWE-generation studies can enhance long-term safety monitoring, planning and investment decisions on MDs (Serrano-Aguilar et al., 2021). Examples are the MAGEC spinal implant devices market withdrawal as a result of long-term RWE collection (Hothi, 2022) and the Food and Drugs (FDA) approval of additional indications for transcatheter hearth valves by using data collected in a post-market valve registry (Dhruva et al., 2018). Finally, several RWE frameworks have been developed to expand the incorporation of RWE in HTA in the last years. Among others, the French Haute Autorité de santé (HAS) 2021 guidelines for the utilisation of RWE in HTA, the National Institute for Healthcare and Excellence (NICE) 2022 RWE framework as a reference for best practices and the Canadas' Drug and Health Technology Agency (CADTH) – The Institut National d'Excellence en Santé et en Services Sociaux (INESSS) – Canadian regulatory authority (Health Canada) joint 2022 national framework for the collaborative application of RWE in HTA (Claire et al., 2024). The aim of this study was thus to provide a detailed overview of published and peer-reviewed practice in post-market assessment of MDs using RWE/RWD. Specifically, we conducted a SR and set the following objectives: (i) to select application papers reporting on RWE/RWD when assessing post-market MDs; (ii) to map the use of RWE/RWD (i.e., evidence type, source, observation time horizon, and aggregation level) throughout MD maturity and type.

2. Materials and methods

2.1 Literature search

A SR was performed using Ovid MEDLINE (1946–2024, February Week 4), EMBASE (1974–2024, February Week 4), and Scopus (2004–2024, February Week 4) databases. We supplemented this search by performing (i) a check on the reference list of the included studies; (ii) a search on Google and Google Scholar (July 2020–February 2024).

Initial searches were carried out in May 2019, updated in July 2020 and in February 2024 to identify the most up-to-date published research. Studies published as of February 2024 were included in the analysis.

A search strategy was developed using both subject headings and free-text terms to capture three main concepts: (i) RWE/RWD; (ii) MD or biomedical technology; (iii) post-market assessment strategy, with a special attention to HTA and health economics analyses. Full details of the search strategy, which was developed in consultation with an expert medical librarian at Oxford University, are provided in the Supplementary file.

2.2 Inclusion and exclusion criteria

For inclusion, studies were required to be full-text publications of peer-reviewed original research published in English and incorporating RWE/RWD into any sort of post-market assessment strategy for an MD working in real-life conditions. Non-English language articles were not included in the analysis due to financial considerations and time constraints related to document translation. Given the exploratory aim of this research, the authors did not apply restrictions in terms of MD types, clinical specialties, comparators, outcomes, or assessment dimensions. Moreover, studies were retained for further analyses irrespective of whether they assessed post-market performance using one-dimensional or multidimensional evaluation strategies, as well as whether they incorporated RWE/RWD alone or in combination with other data sources. As intended in this SR, one-dimensional evaluation studies were defined as studies focusing on a single assessment dimension (e.g., clinical) among those traditionally included into an HTA strategy. Additionally, RWE/RWD were defined as data collected outside the traditional RCT setting (Garrison et al., 2007; Goettsch and Makady, 2016; Makady et al., 2018). We subsequently excluded studies designed as RCTs and/or controlled clinical trials, which were categorized as ‘non-RWE’ studies. A pilot screening of the first 604 articles was independently undertaken by two pairs of authors (EG and FV) and (MV and SM) to develop a common assessment strategy. A first-round screening of titles and abstracts was followed by a second-round screening of full-text articles. The two rounds of screening were independently conducted by two reviewers (SM and EG), and possible discrepancies over the eligibility were resolved by consensus or through discussions with the senior reviewer (MV) until consensus was reached.

Data extraction was undertaken using a pre-designed data extraction form developed in Microsoft Excel (Excel 2016 for Windows, Microsoft Corporation, Redmond, WA) and iteratively refined to capture the key features of the retrieved publications. More specifically, data extracted from each article included: country, MD type and maturity (i.e., adoption/monitoring), MD risk class, innovation complexity, clinical specialty, funding, evidence aggregation level (i.e., monocentric/multicentric study), comparator (if any), population to be treated, patient sample size, time horizon, evidence generation (i.e., source), methodology, and evidence type(s) incorporated into the evaluation strategy. To classify MDs retrieved from literature, the authors were consistent with (i) the updated European risk classification (The European Parliament and the Council of the European Union, 2007); (ii) the classification of healthcare innovations according to their complexity into ‘discrete or simple innovations’, which may not require new training or redesign of organisational process to be used straightway, and ‘fuzzy or complex innovations’, which bring together elements of new technology and organisational (or service) model changes (Atun et al., 2007; Barlow, 2020). Moreover, the parameters of benefit (param) for which the post-market assessment exercise was undertaken, as well as strengths, limitations, key findings, and study outcomes were extracted and categorised. More specifically, the item ‘study outcome’ was codified according to the following algorithm: (i) positive, i.e., statement identifying recommendations to use (or continue to use) the target MD (e.g., cost-effectiveness achieved); (ii) neutral, i.e., statement identifying recommendations to use (or not to use) the target MD, as equal benefits are achieved (e.g., equal costs) versus comparator (e.g., usual care); (iii) negative, i.e., statement identifying recommendations to prefer not to use (or stop the use of) the target MD; (iv) unknown, if recommendations could not be clearly identified as positive/neutral/negative; (v) not identified, if no statement regarding recommendations could be found.

2.3 Data analysis

We employed narrative synthesis to illustrate evidence retrieved from literature. Narrative synthesis, which is based on the application of texts and words to describe literature findings into an appropriate textual narrative, is particularly suitable in cases where a high level of heterogeneity from multiple studies prevents the use of meta-analysis to synthesise evidence (Campbell et al., 2018). This SR was conducted in line with the Preferred Reporting Items for Systematic reviews and Meta-Analyses Protocols (PRISMA) guidelines in an effort to limit any risk of bias and error (Moher et al., 2009). Information about internal validity and study quality of the included studies was extracted and assessed using the Quality Appraisal Checklist (Excellence, 2018) developed by NICE to review HTA evidence on innovative MDs. The QAC checklist is constituted by 14 items measured on a 3-point Likert scale. Moreover, a reduced version of the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) guidelines (Husereau et al., 2013) was employed to extract and appraise economic and/or health economic evidence. Finally, an overall score (i.e., ‘plus plus’, ‘plus’, ‘minus’) was recorded for each study considering the fulfilment of the checklist criteria.

3. Results

The literature search identified a total of 9775 hits, of which 2146 were duplicates (Figure 1). A further 7025 hits were excluded at title and abstract review stage for specific reasons, as outlined in Figure 1. Overall, 604 articles were assessed for eligibility. A detailed review at the full article review stage further excluded 459 articles, primarily because the articles employed non-RWE sources, assessed non-medical device interventions, and focused on different phases of an MD lifecycle than post-market. Finally, a total of 145 primary research articles were included in the SR.

Figure 1. PRISMA flow chart showing the process of paper screening for this review.

The temporal trend of the included articles (Figure 2) highlights, in recent years, the increased interest paid by scientific community to measure post-market performance of new or novel MDs using RWE/RWD sources. Indeed, research articles incorporating RWE sources seem to have kept growing since 2013, with a noteworthy increase after COVID-19 pandemic.

Figure 2. Temporal trend of the included articles identified from the Ovid Medline, Scopus, and Embase databases for each year between January 1990 and February 2024.

3.1 Types of post-market assessment methods

Quantitative methodologies (n = 119/145 [82%]) represented the most common methods used to assess evidence gathered. Only a limited number of quantitative studies reported decision-analytic models (n = 9/145 [6%]) with any form of sensitivity analyses (i.e., PSA vs deterministic). The remaining selected articles employed mixed (n = 13/145 [9%]) or qualitative methods alone (n = 13/145 [9%]).

3.2 Study recommendations

Only nine publications (Webb and Holman, 1990; Tsilimparis et al., 2015; Gregori et al., 2018; Lambers et al., 2019; Bowen, 2020; Tomaiuolo et al., 2021; Goodney et al., 2022; Diedisheim et al., 2023; Ding et al., 2023) reported no statement concerning study recommendations, which were classified as ‘not identified’ (n = 9/145 [6%]). The majority of the selected application papers reported a clear positive recommendation to use or continue to use the MD in routine clinical practice (n = 93/145 [64%]), whereas 18 articles included a study outcome categorised as ‘unknown’, which could not be clearly identified as positive/neutral/negative recommendation (n = 22/145 [15%]). The remaining studies reported negative outcomes (i.e., not use or stop to use the device) (n = 13/145 [9%]) or neutral recommendations on the use of the MD in clinical practice (n = 8/145 [6%]).

3.3 Study characteristics

The use of RWE/RWD (i.e., evidence type, source, time of observation, and aggregation level) was mapped throughout MD maturity and types in each of the included studies (Table 1).

Table 1. The table shows a narrative synthesis of the 145 included studies

^a Mixed funding.

^b Public funding.

Evidence generated by each study was grouped into (i) clinical (124/145 [85%]), which was the most frequently reported; (ii) economic (36/145 [25%]); (iii) social (36/145 [25%]); (iv) organisational (14/145 [10%]); (v) human factor (10/145 [7%]); (vi) ethical (8/145 [6%]). Frequency of reporting performance indicators specifically for each type of evidence (e.g., efficacy/effectiveness for clinical evidence) is shown in Figure 3.

Figure 3. Performance indicators analysed for each HTA dimension incorporated in the included studies.

The frequencies of using RWE/RWD sources among the selected studies (Figure 4) showed that observational prospective/retrospective studies were the most frequently reported (n = 79/145 [54%]), followed by claim/administrative databases (n = 34/145 [23%]).

Figure 4. RWE sources included by the included studies.

Among the selected publications, the total use of each RWE/RWD source increased over time; for instance, the use of claim/administrative databases tripled from 2010 to 2024, with a significant increase after 2020. Studies were also grouped into those including a time horizon less than or equal to 1 year (n = 49/145 [33%]), between 1 and 5 years (n = 34/145 [24%]), and greater than or equal to 5 years (n = 46/145 [32%]). The remaining studies did not clearly specify the time horizon (n = 16/145 [11%]). In terms of aggregation level, studies mostly reported evidence aggregated at hospital (n = 72/145 [50%]) or national (n = 54/145 [37%]) level. Only few studies were conducted at international (n = 12/145 [8%]), regional level (n = 7/145 [5%]). Among the studies that reported patient samples, samples greater than 300 patients (n = 42/145 [29%]) and samples ranging from 100 to 300 patients (n = 36/145 [25%]) were the most utilised. Publications informed by registries (n = 26/145 [18%]) reported more detailed information of the populations to be treated (e.g., age, gender, comorbidities, habits); however, among these, only few studies (Abizaid et al., 2012; Seth et al., 2013; Good et al., 2016; Tasca et al., 2019; Dake et al., 2020) focused on clinically complex populations and elderly patients.

Studies were grouped according to the MD type into (i) therapeutic, – mainly implantable devices –, (n = 74/145 [51%]); (ii) diagnostic (n = 39/145 [27%]), and (iii) surgical (n = 22/145 [15%]) and monitoring (n = 10/145 [7%]). Of the 22 studies assessing surgical devices, 19 reported general surgical procedures involving the specific MD. Furthermore, a noteworthy increase of articles assessing diagnostic devices was observed after 2020 (n = 18/39 [46%]). The most frequently reported clinical specialty was cardio-vascular (n = 53/145 [36%]), while only a small number of studies were identified for the other specialties (e.g., orthopaedics n = 10/145 [7%]). In terms of MD maturity, the monitoring stage was the most frequently reported (n = 106/145 [73%]). A single MD intervention was assessed by the majority of the selected studies (n = 103/145 [71%]), whereas the remaining studies evaluated two (n = 29/145 [20%])or three MDs (n = 5/145 [3%]). Only half of the studies were comparative analysis (n = 76/145 [52%]) that mostly utilised a non-MD intervention (i.e., clinical procedures). Only 28 studies employed another MD as comparator, five papers no intervention and one publication a pharmaceutical intervention. Table 1 shows a narrative synthesis of the included studies.

3.4 Appraisal of the included studies

Quality assessment conducted according to the Quality Appraisal Checklist (Excellence, 2018) revealed considerable heterogeneity. More than half of the selected studies (n = 97/145[67%]) were rated ‘minus’ (low study quality) because few or no checklist criteria were fulfilled. Some of the checklist criteria were fulfilled by 28% (n = 41/145 [28%]) of the selected publications, which were classified as good quality studies (‘plus’). Only eight of the included studies met all or most of the checklist criteria (Briggs et al., 2004; Close et al., 2013; Hympánová et al., 2020; Tsai et al., 2020; Bizzi et al., 2021; Campbell et al., 2021; Bougma et al., 2022; Goodney et al., 2022) and were classified as excellent quality studies (‘plus plus’). A synthesis of the quality assessment is shown in the Supplementary file.

4. Discussion

In this review, we described the incorporation of RWE/RWD into post-market assessment of MDs, and we identified limits, opportunities, and implications of current practices for RWE/RWD generation to guide future research.

Multisource evidence based on non-randomised evidence is increasingly being utilised to inform decisions on the introduction and use of healthcare innovations (Hatswell et al., 2017; Kent et al., 2021), COVID-19 pandemic seems to have further pushed this phaenomenon (Schad and Thronicke, 2022). The review confirmed the increasing reporting of RWE/RWD as the main source of evidence for MDs while highlighting differences in non-randomised evidence generation across time. Claim and/or administrative databases were mostly utilised in the setting of observational studies, associated with multidimensional post-market assessment strategies, and their use tripled between 2010 and 2024, whereas registries were mostly reported by mono-dimensional clinical studies and their use was limited while keeping growing since 2013. The review also revealed that all publications were ‘one shot’ and ‘ad hoc’ studies, as no study was part of a continuous nor periodical post-market monitoring strategy. Moreover, the key limitations identified across all the retrieved publications included: (i) adoption of a narrow approach to the post-market assessment with a focus on a limited number of evidence types, i.e., two dimensions at maximum (n = 122/145 [84%]); (ii) stress on clinical and/or economic evidence gathered in a short/medium time horizon (between 1 and 5 years); (iii) little attention to other relevant evidence dimensions for an MD working in real-life conditions, such as contextual influence and organisational impact; (iv) very limited incorporation of patient perspectives and preference; (v) focus on MDs with a relatively low innovation complexity. Even though in recent years there has been an increasing understanding of the need to seek a broader approach by considering additional parameters of benefit to the traditional ones (i.e., clinical and/or economic), only few publications assessed organisational requirements and/or human factors, which were reduced to usability and/or acceptability excluding a considerable contribution to the assessment in terms of human efficacy and effectiveness (UK Department of Health, 2010; Kelley et al., 2018; Anderson et al., 2019; Manetti et al., 2020a, 2020b). Moreover, the majority of the retrieved studies investigated short (up to 1 year) and/or medium-term (between 1 and 5 years) impact, which may be insufficient to observe longer events related with the MD usage alongside current applications and clinical pathways. Recent public scandals of MDs after their successful introduction into clinical routine practice raised concerns about public health and hopes are addressed to the new European Directive of MDs that should lead to the inclusion of a continuous and systematic life cycle assessment of the devices to overcome limitations of ‘one-shot’ and short-term studies (U.S. Department of Health and Human Services Food and Drug Administration, 2013; Fraser et al., 2020). This review showed that there has been an overemphasis on researching and assessing well-defined, clearly bounded innovations (i.e., relative ‘discrete’ or simple MDs) being adopted by a single organisational unit (i.e., single hospital or team) rather than complex innovations, which bring together technology and organisational or service changes. This SR further revealed substantial heterogeneity in terms of study quality. Firstly, all the 63 publications classified as observational studies usually did not present clear statements of the type(s) of RWE source employed to conduct the study. This may lead to confusion between two separate concepts: data source (e.g., registry) and study design (e.g., observational study), as previously highlighted by Makady and colleagues (Makady et al., 2017). Second, of the 102 retrieved publications including the health economic dimension, only 54 studies specified the decision analytic model used and/or conducted sensitivity analysis. Third, 20 publications did not report the patient sample size. Overall, we documented a general lack of conformity with good practices and little attention to manage decision-making uncertainty. It should be stressed a general lack of inclusion of patient characteristics, preferences, and other relevant user perspectives. Only 22% of the retrieved publications included Patient Reported Outcome Measures (PROMs) in the form of quality-of-life and/or pain assessment data, mainly assessed using standardised generic questionnaires, such as EuroQoL five dimensions (EQ-5D). Few publications (9%) included Patient Reported Experience Measures (PREMs), mostly evaluated through open interviews. However, the use of such source of evidence increased after the COVID-19. To our knowledge, this review is one of the first attempt to systematise key features, empirical uses, and quality of RWE/RWD studies across time in response to the increasing attention paid by scientific community when assessing post-market MDs. A strength of this study is that it is consistent with the PRISMA guidelines and followed its checklist to pilot reporting extracted features from the included studies. This SR also revealed that all publications were ‘one shot’ studies and there was huge heterogeneity in terms of evidence generation, MD type and clinical application, as well as study quality. Furthermore, this study provided the chance to gain a snapshot of the effect of the COVID-19 pandemic over the use of RWE and RWD to assess post-market MDs. As a result, the pandemic has provided a boost to the adoption of RWE for MDs assessment, as the number of publications has significantly increased after 2020. This probably occurred as a response to the limited time and resources available for conducting RCTs ‘ex-novo’ in pandemic moments. An uprising in diagnostic technologies focused studies was observed, jointly with hospital level papers. Hospital level studies were probably preferred as national and international exchanges have mostly been interrupted during COVID-19 pandemic. Two articles dedicated to COVID-19 specific technologies were included in this SR (Tomaiuolo et al., 2021; Mafi et al., 2023). Further studies should be carried out to gain a full picture of the effect of the pandemic on the topic.

Potential limitations include the English language, which affected the geographical distribution of the results, as most of the included studies come from English speaking countries (e.g., UK and Canada). Non English-speaking countries experiences should be further explored, since different post-marketing HTA RWE adoption could emerge. For example, Asian countries recognise the value of and already use RWD/RWE in HTA, given the lack of relevant pre-marketing data from clinical trials for the region. These countries rely on RWE to adjust prices and reassess funded technologies or make reimbursement decisions based on RWE several years after market entry (Lou et al., 2020). The grey literature encompassing non-peer-reviewed publications, such HTA reports, was also excluded, which may limit the comprehensiveness of the use of RWE in practice. Therefore, for some innovations regional and/or national bodies act as ‘gatekeepers’ to the health system by gathering evidence and produce HTA reports written in local languages (e.g., French, German etc.). In addition to this, the authors faced with substantial publication biases. The review confirmed that almost all post-market studies funded by private bodies reported a clear positive outcome of the study. Health economic analyses are generally not reported by HTA bodies and come from private funders (Turchetti et al., 2017). The authors expect that such analyses are only published when the outcome is positive (i.e., publication bias). The previous limitations prevent the authors to take a definitive picture of the current practices in post-market assessment of MDs and make comparisons across regions. After the pandemic, the interest and the publications related to Machine Learning Models and Artificial Intelligence as components of existent or new MDs increased but findings related to post-market assessment of these type of MDs are yet at the beginning.

5. Conclusions

The use of non-randomised evidence is growing steadily when assessing post-market MDs (Crispi et al., 2019; Kent et al., 2021). Indeed, RWE/RWD are particularly relevant for MDs because of their peculiarities, such as user-dependency. In fact, uncertainties and limitations concerning evidence on safety, efficacy/effectiveness, and cost-effectiveness, as well as the rate of innovation uptake, are intrinsically linked to the unique challenges of MDs compared to traditional health technologies, such as drugs and pharmaceuticals. To the best of our knowledge, despite the large use of non-randomised evidence when assessing MDs, empirical studies and reviews focusing on a specific device and/or a target clinical area have already been published, however a comprehensive picture on the current practice and the implications of using RWE to inform policy decisions is currently lacking. In this sense, our study is the first in its kind to provide a holistic picture of how non-randomised evidence has been used when assessing MDs working in real-life conditions. This review seeks to provide an empirical-based foundation for the use of RWE/RWD in adopting, monitoring, and assessing post-market performance of new or novel MDs alongside clinical routine practice. Our findings led the authors to draw some policy implications addressed to governments and healthcare organisations.

Firstly, the review highlighted the need for a shift from ‘ad hoc’ and ‘one-shot’ studies to monitoring systems that allow the continuous performance assessment of post-market MDs. Indeed, the variability in the quality of care, access, equity, and the financial aspects related to the use of MDs across countries, regions or hospitals and health organisations can be observed and reduced if the monitoring system is continuous and systematic using a benchmarking approach. This can lead to a continuous RWE/RWD generation alongside clinical routine practice, prevent public safety scandals, as well as ensure a fairer allocation of health resources. Hence, we recommend including MD performance indicators with a population-based perspective into wider performance evaluation systems at healthcare pathway level (see, for instance, the Italian experience of measuring the performance path (Gunn et al., 2021)). Indeed, HTA and performance evaluation systems can be considered as naturally linked as they are both data-driven systems adopting indicators to report on several healthcare organisations' dimensions performances and they inform policy-makers' decisions (Nuti et al., 2018). Specifically, Figure 3 shows that commonality exists between performance evaluation systems and HTA dimensions. However, further studies should be conducted to explore the relationship between HTA and performance management in depth. Secondly, the review highlighted that few of the included studies deals with a medium-long time horizon (i.e., greater than 5 years). It should be stressed that the adoption of a short time window may be insufficient to observe longer events related with the MD usage alongside current applications, especially for implantable devices, whose side effects on safety and effectiveness are little known during adoption. For specific types of MDs (e.g., TAVI), healthcare organisations activated devices' registries and traceability systems, however no evidence in terms of iterative or periodical assessment has been found in the literature retrieved.

Thirdly, health managers and policy-makers might finance more multidimensional assessment studies, focus on more innovative MDs that require significant organisational changes into current frameworks, as well as promote more publicly funded RWE/RWD studies. Among the dimensions of interest, environmental and assessment is gaining particular attention, since healthcare systems account for a substantial proportion of global carbon emissions and contribute to wider environmental degradation (Pinho-Gomes et al., 2022). This specifically referred to MDs, given their well-known environmental impact (Sousa et al., 2021). Encouraging public research on post-market assessment/monitoring is desirable not only to increase knowledge into MDs' routinary use and applications but also to generate independent evidence that ensure more transparency of the results obtained. Indeed, studies funded by public bodies can contribute to generating evidence for MDs' ‘non-use’, which is currently lacking. Furthermore, a research agenda has been identified for research scholars aiming to increase efficacy and quality of evidence generation in post-market phases with a population-based approach. Future research is needed to close the gaps highlighted by this review. In particular, scholars are asked to (i) close the evidence gap between RCT and real-world by continue to conduct real-life assessment studies; (ii) shift their research efforts on more complex or fuzzy boundaries innovations involving multiple changes to healthcare practices and targeted at service and/or professional role redesign, this also considering the arrival of particularly innovative technologies based on Artificial Intelligence and on Machine Learning Models; (iii) incorporate the personal value in future RWE/RWD studies, i.e., the value determined by the fit between the study outcome and the individual user including patient value; (iv) incorporate the personal value in future RWE/RWD studies, i.e., the value determined by the fit between the study outcome and the individual user including patient value (Gray and El Turabi, 2012; Gray et al., 2017; Li et al., 2019; Gunn et al., 2021). (v) generate more multidimensional evidence on both MDs' use and ‘non-use’ and also consider novel relevant MDs' dimensions (e.g., environmental and sustainability dimension); (vi) consider also to provide evidence on the last stage of MDs maturity, such as obsolescence and replacement, which are under-investigated by scientific literature which can be really relevant in terms of disinvestment; (vii) examine the relationship between RWE post-market HTA and other systems or theories that present common and innovative elements to learn from and that can support further MDs' HTA development.

Although it might be an issue covered by grey literature and reports, it could be relevant to have an overview of the MDs that are disinvested, replaced, or re-adopted/re-allocated in other clinical settings.