Although stochastic analysis has become the accepted standard for decision analytic cost effectiveness models, deterministic one-way sensitivity analysis continues to be used to meet the needs of decision makers to understand the impact that changing the value taken by one specific parameter has on the results of the analysis. However, there are a number of problems with this approach.

We review the reasons why deterministic one-way sensitivity analysis will provide decision makers with biased and incomplete information. We then describe a new method - stochastic one-way sensitivity analysis (SOWSA), and apply this to a previously published cost effectiveness analysis, to produce a stochastic tornado diagram and conditional incremental net benefit curve. We then discuss how these outputs should be interpreted and the potential barriers to the implementation of SOWSA.

The results illustrate the shortcomings of the current approaches to deterministic one-way sensitivity analysis. For SOWSA, the expected costs and outcomes are captured, along with the sampled value of the parameter and these are linked to the probability that the parameter takes that value – which can be read off the probability distribution for the parameter used in the stochastic analysis. From these results it is possible to gain insights into probability that a parameter will take a value that will change a decision.

Although a well-used technique, one-way deterministic sensitivity analysis has a number of shortcomings that may contribute to incorrect conclusions being drawn about the importance of certain parameter values on model results. By providing fuller information on uncertainty in model results, it is hoped that the methods here will lead to more informed decision making. Although, as with all developments in the presentation of analytic results to decision makers, care will be required to ensure that the decision makers understand the information provided to them.

Economic models contain several parameters ordinarily subject to uncertainty. Unlike most other model parameters, costs can constitute numerous distinct components. For example, a surgical intervention can require a variety of disposables and reusable equipment. A micro-costing output may be disaggregated or presented as a total cost. Uncertainty could be applied to individual cost components or to total cost. We aimed to explore how disaggregation of cost data may impact on uncertainty using a case study.

A set of simulations using hypothetical scenarios were developed with uncertainty set at ± 20 percent. The simulations investigated the impact of number of cost components, balance between components, and correlation between them. A cost-utility model from an assessment of robot-assisted radical prostatectomy was analyzed; procedure cost was divided into 32 individual cost components or treated as a total cost.

Based on five equal cost components, uncertainty reduces from ± 20 percent for correlated variables to ± 9 percent for uncorrelated variables. With increasing numbers of uncorrelated cost components, the uncertainty in the total cost decreases markedly. The uncertainty around total robot-assisted surgery procedure equipment costs was ± 19.7 percent when components were correlated and ± 9.4 percent when uncorrelated. The impact on uncertainty in the incremental cost effectiveness ratio (ICER) was negligible but the ranking of parameters in the univariate sensitivity analysis changed.

Analyzing uncertainty by aggregated or disaggregated costs can have implications for presenting uncertainty in costs to decision makers. Applying uncertainty to aggregated costs essentially implies that variation in the cost of individual components is perfectly correlated. By disaggregating cost components they are being treated as uncorrelated, which can substantially reduce uncertainty in the total cost. In this case study we found that although the reduction in uncertainty could be clearly seen in the cost parameter, it had a negligible impact on uncertainty in the ICER.

For the last decades, health technology assessment has been strongly promoted in order to provide evidence-based rather than eminence-based healthcare. However, when it comes to implementing interventions that are based on processes and behavior rather than products, importing evidence-based interventions is likely to fail because it ignores the strong influence of contextual factors. In a recently started research project, an alternative approach is tested.

The project aims at improving identification and support of children who have parents with a mental illness in an Austrian region (Tyrol). A theory- and stakeholder-driven approach has been designed in order to co-develop, implement and evaluate practice approaches for improving the situation for children. The former addresses the questions whether, how and why suggested practices may work and the later brings together evidence and practice to develop interventions that are feasible and take the regional context and service settings into account.

Based on evidence from various sources (literature, international and local stakeholder interviews), theories that describe the key mechanisms of action to influence outcomes for children have been developed. A regional and interdisciplinary group of practitioners and people with lived experience has been established to facilitate the development of the theory of change and practice approaches for the region and local service settings. The final practice model is then monitored in the respective organizations with implementation support from the research team.

Theory-driven and co-designed-based approaches are a feasible alternative to ‘off-the-shelf’ evidence-based practices for supporting decision makers in implementing complex interventions. However, they require a broad variety of skills within the research team as well as willingness to accept uncertainties of the final outcomes produced, which can also be of risk to funders. The ongoing project will demonstrate whether the developed practices will be implemented successfully and result in benefits for the affected children.

Australia has a well-developed HTA system for approving, funding and disinvesting in medical services, but how does it cope when it assesses new or existing services when health system frameworks, policy and legislation do not keep pace? This presentation will present a number of case studies where HTA methods have adapted to suit emerging health needs of the Australian community.

Australia's HTA system has been adapted over recent years to allow it to perform HTA on novel services that do not fit into its standard HTA pathway to public funding. There has also been an increasing number of assessments where HTA has been unusually sponsored by the funder to assess priority health care needs.

More Medicare funding for addiction and sexual health consultation services are an example of a novel HTA which led to more public funding for these services. Limited evidence for these services was available and there was difficulty in demonstrating that increased public funding would lead to better outcomes for patients. A range of techniques, such as examining real world data and stakeholder views were partnered with HTA to assess these services. Currently, HTA is being utilised to assess a novel medical treatment known as the anti-CD19 chimeric antigen receptor T cell (CAR-T) therapeutic process. This is a complex, non-standard HTA which encompasses aspects of the Australian hospital funding system, Medicare and the Pharmaceutical Benefits Scheme and requires an adapted HTA process to assess evidence across a range of funder systems.

Australia's well established HTA system has adapted to become more agile to suit emerging health care needs for a range of interest groups, the government, sponsors and consumers. Consultation with stakeholders and the community have assisted in developing and refining these new processes.

Mental health disorders and their treatments produce costs and benefits in both healthcare and non-healthcare sectors. The latter one is often referred to as inter-sectoral costs and benefits (ICBs). Limited research is available on the inclusion of these inter-sectoral costs and benefits (ICBs) in economic evaluations. In this study, we focus on the identification and classification of ICBs of mental health-related interventions within the criminal justice sector in a broader European context. This study was conducted as part of the PECUNIA-project, which aims to develop new standardized, harmonized and validated methods and tools for the assessment of costs and outcomes in European healthcare systems. The aim of the study is to further conceptualize an internationally applicable list of ICBs of mental health-related interventions in the criminal justice sector. Additionally, we aim to facilitate the inclusion of ICBs in economic evaluations across EU by prioritizing important ICBs.

Data was collected via a systematic literature search on PubMed and PsychINFO. Additionally, a grey literature search was carried out in six European countries. In order to validate the international applicability of the list and prioritize the ICBs, a survey was conducted with an international group of experts from the criminal justice sector.

The literature search identified ICBs and resulted in a comprehensive list of items. A multi-dimensional list was constructed, distinguishing between costs as consequence of crime, and costs in response to crime. Based on the expert survey, the international applicability of the list was validated and the most important ICBs from the economic perspective were identified.

This study laid further foundations for the inclusion of important societal costs of mental health-related interventions within the criminal justice sector. More research is needed to facilitate the greater use of ICBs in economic evaluations.

This study aimed to reach patients using different languages while providing an opportunity to enter symptoms in their everyday language text besides medical expressions of symptoms.

Named entity recognition (NER) techniques, based on natural language processing (NLP), were applied to develop a language independent predictive model. The research was based on extracting symptoms entered to the system by patient using NER method of NLP. In order to implement the system, python was used while pre-processing the data and string similarity function was used to estimate similarity with disease symptoms. Two sets were used for classification, one including only symptoms, and the other the matching diseases. Four thousand two hundred and eighty different symptoms were processed for the corresponding 880 diseases.

Each user symptom had a similarity score for each symptom in all diseases. Top N results with highest similarities were chosen from this list. The final N results are matched with diseases. According to these results, matched diseases were ordered in terms of the percentage of matched symptoms in the disease's symptoms. Extracted terms were implied as an input of the model and analysed for a matching diagnosis where an accuracy of 83 percent was accomplished when it is tested and compared using Mayo Clinic data for specific foreign languages other than English.

This language independent online diagnostic tool is a solution for both personal and clinical use and provides maintainable, updatable and more reliable diagnostics. This tool is particularly relevant today, with global mobility growing at a rate faster than the world`s population. We aim to upgrade the system by adding speech recognition and engaging it with the background (if available, electronic health records) of the patient.

Robot-Assisted Surgery (RAS) has been available in Australia since 2003. There are 50 da Vinci RAS systems in Australia (18 in NSW and 12 in Victoria) with most in the private hospital sector. In Australia the capital cost of a da Vinci RAS system is up to AUD 4 million (USD 2.8 million), which excludes annual maintenance fees of AUD 250,000 (USD 175,000) and consumable costs of AUD 3,500 (USD 2,450) for each procedure.

The NSW Ministry of Health and Victorian Department of Health and Human Services commissioned a health technology assessment (HTA) to explore the benefits, risks and economic implications of surgical robotics, which involved a review of the peer reviewed literature, a cost benefit analysis of public sector patients who received RAS and broad stakeholder consultation to document current perspectives on RAS applications.

RAS is as safe and effective as other surgical modalities when performed by sufficiently skilled surgeons, although evidence generally comes from small studies with limited follow-up time and few studies report long term mortality, morbidity or patient-reported outcomes. Comparative benefits of RAS are uncertain as most studies conclude little or no difference in procedure related or functional outcomes. While RAS reduces length of stay, which offers patient and health system benefits, this is insufficient to fully offset high capital and consumable costs currently charged to Australian providers. Government and clinical stakeholders identified that establishing an RAS service requires consideration of important factors, including: i) Governance is critical; ii) Higher case volumes may improve financial viability; and iii) a need for state-wide/national standards for surgeon training and credentialing.

RAS is as safe and effective as other modalities when performed by skilled surgeons. However, uncertainty remains around long-term outcomes and clinical and cost effectiveness. An accredited training program, monitoring and evaluation will be critical to ensure outcomes data inform ongoing evidence assessment and government policy and investment.

The Brazilian Unified Health System (SUS) is known worldwide for ensuring universal assistance to citizens, which includes the supply of medicines free of charge. Biological drugs consume about 40 percent of the public budget for pharmaceutical services in the SUS and Rheumatoid Arthritis (RA) is the largest consumer of these resources, serving about 110,000 patients. Since 2002 there has been a clinical guideline for the care of patients with rheumatoid arthritis in Brazil, currently providing 10 biological drugs for treatment of RA. The objective of this study is to present data about the provision, expenditure and profile of users of biological drugs for rheumatoid arthritis in SUS.

Retrospective and exploratory study, using administrative data regarding the purchase and consumption of biological drugs Infliximab, Etanercept, Adalimumab, Rituximab, Abatacept (intravenous and subcutaneous), Tocilizumab, Golimumab and Certolizumab pegol for the treatment of RA between 2012 and 2017 in SUS.

There was an expenditure of approximately USD 421.7 million from the Brazilian Ministry of Health with the supply of about 2 million pharmaceutical units of biological drugs for treatment of rheumatoid arthritis, 79 percent of them destined for female users and 89.2 percent for the 40-69 age group. The M05.8 and M.06.0 codes of the International Classification of Diseases (ICD-10) were the most prevalent among the arthritic population served. Adalimumab and Etanercept accounted for 68.3 percent of total expenditure. A reduction in the use of these medicines were observed after the availability of new drugs for the treatment of the disease between 2014 and 2017.

Brazil is one of the largest consumers of biological medicines in the world. The use of real-life data allows monitoring trends and costs of the use of these drugs as well as changes with the entry of new therapies and biosimilar medicines.

Stroke is a major contributor to mortality, disability and long-term use of healthcare services. As for all chrono-dependant conditions, clinical results are associated with timely access to appropriate care. Thrombectomy (EVT) is an effective treatment for large vessel occlusions, but can only be provided in highly-specialized centers by experienced personnel. We sought to develop a framework to aid decision-making on the appropriateness of opening new EVT centers in Québec, Canada.

Data sources included provincial administrative healthcare databases, population density statistics, field evaluation of Québec's four existing EVT care networks, and literature review concerning structural and performance criteria for EVT centers. We consulted EVT clinical teams, interdisciplinary stroke experts, patients, professional association representatives, healthcare managers and decision-makers.

Access to EVT is suboptimal in all 17 regions of Québec, with virtually no access in remote areas. Results of key performance indicators indicated favorable treatment delays after arrival at the EVT center. However, door-to-needle and door-in-door-out times were long for patients transferred from non-EVT centers. High use of ambulances indicated the potential to transport patients to the most appropriate center. In light of ‘real world’ results and other sources of information, the need for a new EVT center should consider the following criteria: sub-optimal EVT access within the region; transport time to an existing EVT center >1 hour; expected patient volume within 2 hours of transport; impact on volume of existing programs; availability of long-term financial support; availability of a critical mass of neurointerventionists, vascular neurologists, and neurosurgeons; demonstrated quality of stroke care; and, presence of a stroke unit.

The triangulation of literature, clinician experience and the Québec context enriched the evaluation process. Furthermore, this facilitated the development of a framework that was broadly applicable across regions to the real-world setting of decision-making in a complex system of care.

The objectives of this study were to systematically review published research on the relationship between nursing staff coverage, care hours, and quality of care (QoC) in long-term care (LTC) facilities; and to conduct a real world evidence (RWE) analysis using Alberta real world data (RWD) to inform policy makers on whether any amendments could be made to current regulations.

A systematic review (SR) of research evidence published between January 2000 and May 2018 on the relationship between nursing staff coverage, care hours, and QoC in LTC facilities was conducted. Panel data regressions using available RWD from Alberta, Canada, were performed to assess associations between nursing care hours and LTC outcomes. Outcomes of interest included quality indicators related to resident outcomes, hospital admissions, emergency room visits and family satisfaction. Nursing care hours considered in SR and RWE analysis included those provided by registered nurses (RNs) and licensed practical nurses (LPNs).

The SR found inconsistent and poor quality evidence relevant to the questions of interest, indicating a great uncertainty about the association between nursing staff time and type of coverage and QoC. Although some positive indications were suggested, major weaknesses of reviewed studies limited interpretation of SR results. RWE analysis found that impact of care hours on LTC outcomes was heterogeneous, dependent on outcome measurements. There was evidence that total staff, RN, and LPN hours had positive effects on some resident outcomes and magnitude of effect differed for different nursing staff.

No definitive conclusion could be drawn on whether changing nursing staff time or nursing staff coverage models would affect residents’ outcomes based on the research evidence gathered in the SR. RWE analysis helped to fill a gap in the available published literature and allowed policy makers to better understand the impact of revising current regulations based on actual outcomes.

When conducting a Network Meta-Analysis (NMA) for a Health Technology Assessment (HTA), the submitting company typically will have access to Individual Patient Data (IPD) from their own trials, but only aggregate data (AgD) for the comparator. In this case, they can re-weight the IPD so that the covariate characteristics in the IPD trials match that of the AgD trials, using the increasingly popular method of Matching-Adjusted Indirect Comparison (MAIC).

We carried out a simulation study to investigate this method in a Bayesian setting. We simulated three IPD trials comparing treatments A and B (AB-IPD trials), and one aggregate data trial comparing treatments B and C (BC-AgD trial). We investigated two options of weighting covariates: 1. all three studies are weighted separately to match the BC-AgD trial (MAIC Separate Trials). 2. patients are weighted across all three IPD studies to match the BC-AgD trial, but the NMA still considers each trial separately (MAIC Pooled Trials). We compared the results of the MAIC to a standard NMA and a mixed IPD/AgD NMA. We applied these methods to a network of treatments for multiple myeloma.

MAIC can provide more accurate estimates of the relative treatment effects than a standard NMA in the BC-AgD trial population. However, MAIC may decrease the accuracy of the relative treatment effects in the overall population. Treatment rankings were unchanged when applying MAIC to the multiple myeloma network.

MAIC is beneficial as a sensitivity analysis to demonstrate that results hold across patient populations. If there is a difference in relative treatment effects attributable to population imbalances, then it is useful to be able to quantify this difference. However, we recommend using either a standard NMA or a mixed IPD/AgD NMA for the base case analysis, given the potential bias that can arise in an MAIC.

Efforts to harmonize health technology assessment (HTA) processes and methods across Europe are currently intensified. In this context, the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach has been proposed as a “common ground” in joint HTAs. However, GRADE has been primarily developed to support authors of clinical guidelines. Therefore, it is unclear whether HTA reports based on GRADE are compatible with the methods currently applied by European HTA organizations.

We contrasted IQWiG's methods paper and publications by the GRADE Working Group with regard to the following domains: 1) risk of bias (RoB) assessment 2) prerequisites for “greater benefit” (assuming that IQWiG's “greater benefit” corresponds to a GRADE assessment of at least low certainty and a small important effect) and 3) consideration of non-randomized studies (NRS). We present illustrative differences and highlight similarities.

Overall, RoB assessments are very similar under both approaches. However, we identified several important differences. In case of very severe publication bias, IQWiG methods preclude drawing a conclusion, whereas GRADE requires only downgrading the certainty of evidence while still allowing for a conclusion on effect sizes. Secondly, IQWiG generally requires a statistically significant effect for a “greater benefit”, while GRADE does not (statistically non-significant effects would only necessitate downgrading the certainty of results for imprecision). Another difference is that in general, NRS are not included in IQWiG assessments when randomized studies (RS) are available and thus possible. In contrast, preliminary GRADE guidance recommends considering NRS in addition to RS when the RS evidence is of low or very low certainty.

While GRADE and IQWiG's method share some similarities, our exemplary analysis shows that there are some notable differences. Therefore, GRADE should not be used “out of the box” for European HTAs. To foster further discussion, more research (including a comprehensive comparison of methods and an analysis of resources for adaptation) is needed.

Intention to treat (ITT) is a gold standard strategy to analyze the results of randomized controlled trials (RCTs). ITT analysis has been considered a methodological indicator of the quality of clinical trials. The extent to which the use of ITT is related to the treatment effects observed in RCTs has not been rigorously explored. Therefore, the main objective of this study was to determine the association between biases related to attrition and missing data and the use of intention to treat principle, and changes in effect size estimates in RCTs.

This was a meta-epidemiological study. A random sample of RCTs included in meta-analyses was identified. Data extraction including assessments of the use of intention to treat principle, missing data and drop-outs was conducted independently by two reviewers. To determine the association between biases related to attrition, missing data, and the use of intention to treat and effect sizes, a two-level analysis was conducted using a meta-meta-analytic approach.

Three-hundred and ninety-three trials included in 43 meta-analyses, analyzing 44,622 patients contributed to this study. From these, 134 trials (34.1%) used ITT and 218 (55.5%) did not use ITT. Trials which did not use the ITT principle, or which were assessed as having an inappropriate control of incomplete outcome data (based on the Cochrane risk of bias tool) tended to underestimate the treatment effect when compared with trials with adequate use of ITT (ES= -0.13; 95%CI -0.26, -0.01) or trials which were assessed as having an appropriate control of incomplete outcome (ES= -0.18; 95%CI -0.29, -0.08).

Our results suggest that when evaluating risk of bias of primary RCTs, systematic reviewers should pay attention to these biases since they could underestimate treatment effects. Systematic reviewers should perform sensitivity analysis including trials with low risk of bias in these domains.

Most new oncology therapies are studied in the advanced/metastatic setting. However, there is an increasing focus on earlier stage disease. Nevertheless, measuring Overall Survival (OS) in neo-/adjuvant therapy trials can be very challenging due to the increased life expectancy and the confounding effects of subsequent treatments. Thus, their primary endpoints tend to be surrogate survival metrics (e.g. metastases-free survival). This research aims evaluates the health technology assessment (HTA) acceptability of such endpoints through recent neo-/adjuvant HTA assessments.

The European Medicines Agency (EMA) website was screened for any neo-/adjuvant oncology therapies approved (1 January 2013-22 October 2018) and any corresponding publicly-available assessments by HTA bodies (NICE, SMC, IQWiG, G-BA, CADTH, PBAC, HAS) were identified and key data extracted.

Six neo-/adjuvant therapies have received marketing authorization by the European Commission (EC). These six have been on the market for an average of 8.9 months (range: 0.9-39.3 months, median: 3.3 months). In four of the six, the pivotal trial primary endpoints were measures of relapse-/disease-free survival, (others: pathological complete response and PFS/OS co-primary). Only one had mature OS data available at EC-approval. Four of the six therapies had received at least draft guidance by an HTA body, encompassing 11 HTA assessments in total (4: NICE, 2: IQWiG, HAS; 1: SMC, CADTH, G-BA). Only two of 11 (18%) were positive outcomes (both NICE), the remaining nine were negative.

Oncology therapies are increasingly receiving regulatory approval in the neo-/adjuvant setting. However, their pivotal trials are frequently powered to show benefits in disease-/metastases-free survival. Whilst sufficient for regulatory approval, translating this to favorable HTA decisions has been more challenging. Clearly establishing linkages between surrogate survival metrics and OS alongside measuring metrics that clearly portray patient benefits (e.g. time to symptomatic progression) could improve HTA-acceptability. Further, some payers allow for temporary reimbursement whilst additional evidence is generated (e.g. Cancer Drugs Fund in England).

The introduction of fast-track licensing strategies increases the approval of anti-cancer drugs with ambiguous benefit-risk profiles. Thus, in many instances there is lacking evidence about overall survival (OS) at the time of marketing authorisation. Our objective was to monitor and characterise therapies with ambiguous benefit-risk profiles and identify any post-approval updates on median OS after at least three years of approval by the European Medicines Agency (EMA).

We included all originator anti-cancer drugs with initially ambiguous benefit-risk profiles that received marketing authorization from the EMA between 1 Jan 2009 and 31 May 2015. Our monitoring timeframe was at least three years after EMA-approval. To identify study updates, the following three sources were included: clinicaltrials.gov, European Public Assessment Reports (EPARs), and PubMed.

In total, we identified 102 eligible approval studies. Out of these, a negative difference in median OS or no information was available in forty-three (42.2%) instances. During monitoring, eleven updates with accessible information on median OS could be identified. Including monitoring results, there are still thirty-two remaining therapies (31.4%) where no or negative information (n = 27 [26.5%] and n = 5 [4.9%], respectively) regarding median OS was present at least three years after EMA approval.

One-third of oncology drugs with ambiguous benefit-risk profiles failed to demonstrate a survival benefit even several years following marketing authorization. Systematic and transparent post-approval monitoring mechanisms will be of high relevance to assure a clinically relevant patient benefit, since the trend towards faster access to medicines with uncertain benefit is increasing rather than declining.

Following marketing authorization in Spain, new medicines are assessed by the Inter-Ministerial Pricing Commission for Pharmaceuticals (CIPM), which provides reimbursement recommendations with a maximum ex-factory price. However, there are 17 autonomous regions, which can make distinct reimbursement decisions. To drive consistency, the Spanish Agency for Medicines and Health Products has issued national Therapeutic Positioning Reports (TPRs) for new medicines since 2012. Since November 2017, CIPM recommendations have been published monthly, giving the opportunity to analyze the impact of TPRs on the speed and outcome of CIPM decisions, which this research evaluates.

Publicly-available CIPM and TRP decisions were identified from www.msssi.gob.es and www.aemps.gob.es, respectively. Marketing authorization dates were identified from www.ema.europa.eu or www.aemps.gob.es (10 March 2007-11 February 2018). Pearson's chi-squared and Mann-Whitney U statistical tests were performed using R.

One hundred and ninety-three drug-indication pairings with an associated TPR were identified. The majority (62% [120/193]) were recommended as alternative treatment options with only 19 percent (36/193) deemed to be superior and 19 percent (37/193) not recommended. One hundred and eight CIPM recommendations were identified across seven monthly reports, issued a mean of 12.2 months after market approval, 59 percent (64/108) were positive and 41 percent (44/108) were negative recommendations. There were 34 drug-indication pairings with both CIPM and TPR recommendations available. Of these, 24 percent, 56 percent and 21 percent had TPR outcomes of ‘superior’, ‘alternative’ and ‘not recommended’, respectively and 71 percent and 29 percent had positive and negative CIPM outcomes. Drug-indication pairings with ‘negative’ TPRs were significantly more likely to have negative CIPMs than those with either ‘alternative’ or ‘superior’ TPRs (71% vs. 19%, respectively, χ2 = 5.16, p = 0.02) and were more likely to experience significantly longer delays to CIPM recommendation (23.9 vs. 13.5 months, respectively, U = 50, p = 0.03).

Drug-indication pairings with ‘positive’ and ‘alterative’ TPR outcomes are associated with significantly better and faster CIPM recommendations than those with ‘not recommended’ TPR outcomes

The use of digital technologies in healthcare systems (digital health)– such as electronic health records and telehealth – can improve primary care (PC). However, integration of digital health can be constrained/impaired and/or facilitated due to several factors. We propose an integrative framework for classifying the factors that could favour or limit digital health integration in PC in order to guide the identification of strategies that could be helpful for technology promoters, managers, clinicians and researchers.

Based on a systematic review, our framework includes seven categories to classify the main opportunities and threats to digital health integration in PC: technological; individual/interpersonal; professional; organisational/institutional; ethical/legal; sociopolitical; economical. We consulted a panel of researchers, managers, clinicians, and citizens/patients in a scientific meeting regarding the main opportunities and threats to the integration of digital health in PC. We performed a content analysis of the reported factors according to the framework.

Technological factors such as maturity, interoperability and ease of use were often mentioned as key conditions for digital health integration. Individual and interpersonal factors such as depersonalisation and digital literacy were seen as threats. The impact on workload and shared responsibility were threats at the professional level, whereas silos and change management were noted as organisational threats. Current policies and social trends favored digital health. Threats regarding privacy and confidentiality were mentioned at the legal/ethical level. The possibility to reduce costs and sharing of benefits were noted as opportunities at the economic level.

Knowing these multidimensional conditions, perceived as either threats or opportunities depending on the context of each PC setting, is essential to inform decisions, from strategic planning to evaluation. Our integrative framework allows a simple classification of opportunities and threats that can guide the development and implementation of tailored strategies favouring the integration of digital health in PC.

Mobile Health (mHealth) apps offer potential to promote greater public engagement in health, improve efficiency and open up new care pathways and models of care. However, the volume and heterogeneity of apps has led to uncertainty and lack of standardization around app definitions. Some mobile apps carry minimal risks to consumers, but others can carry significant risks. Work has been carried out to develop a framework for assessment (for example, for the NHS app library [beta version]). We discuss work helping to inform a preliminary framework of categorizing mHealth apps for proportionate assessment and validation, and the challenges involved.

A literature review was carried out to identify different types of categorizations used to define health apps and the most important dimensions for their assessment. A taxonomy of apps and a process for routing them towards appropriate methods of evaluation was developed through iterative review, discussion and refinement.

Fourteen types of mHealth apps were established which were categorized by app function and by the potential risk involved with use. Subsequently, this research suggested a method of routing apps towards the most appropriate and proportionate method of evaluation, by using four example dimensions of impact (population size, disease burden, priority of clinical condition, and innovation), and four levels of risk.

The outcome of an evaluation framework should be to enable healthcare professionals and patients to select and use safe and effective mHealth apps with greater confidence. A preliminary taxonomy and method of routing apps towards appropriate assessment are presented. Both need larger scale discussion, iterative testing and refining. This research faced significant challenges, including a high volume of heterogeneous apps with poorly standardized app definitions and associated nomenclature.

Smart medical devices can empower elderly to live independently in their familiar surroundings. To enhance their dissemination, they have to be shown to be cost-effective. Economic studies evaluating such technologies are missing or are criticized for their low quality. There are several challenges in the evaluation of smart medical devices, including their complex nature and innovative character. The question arises: how can evaluations elicit the benefits and cost-effectiveness of smart medical devices. This research has the aim of outlining challenges and demands on the evaluation of smart medical devices.

The embedding of the technology in existing structures can influence the effectiveness of the technology. By comparing such a technology with a regular intervention, learning effects have to be considered. Regular modifications and further developments of these technologies can complicate the traceability of the effects. Complex cause-effect relationships with possible interactions arise that are difficult to quantify and express in standardized endpoints, utilities or monetary values. Demands on the evaluation of smart medical devices have been explored with literature reviews and scenario techniques using the example of intelligent rollators.

It is important to apply mixed-method approaches not only in the clinical but also practical setting and conduct observational as well as qualitative studies. Potential users, their relatives and care personnel should be involved in the evaluation of intelligent rollators and attention should be payed to subjects with disabilities. Prospective studies should be conducted at different stages along the lifecycle of the technology. A conceptual model should be developed and evaluated as well as adapted on a regular basis.

The research shows the need to adapt common methods used in economic evaluation to the characteristics of smart medical devices. As a next step, a framework for the economic evaluation of such technologies within the scope of Health Technology Assessment is developed based on these demands.

There are more than 320,000 accessible health apps, with the most downloaded of those related to physical exercise and weight control. However the initiatives for their validation address only partial aspects of the evaluation. The EVALAPPS project aims to develop an assessment tool for overweight and obesity management apps, based on the evaluation of efficacy, effectiveness and safety. In the present phase of the project, the team is co-creating the assessment tool considering both the evidence and the expertise of professionals (co-creation process).

Proposed co-creation methodology includes: 1) a modified Delphi process for selecting the assessment criteria. Criteria were identified through a) an exhaustive review of the criteria used by several mHealth assessment tools and b) a systematic review of efficacy, safety and effectiveness criteria used in mHealth interventions that assess overweight and obesity management. 2) a co-creation session using “Design Thinking” techniques for defining the final content and appearance of the tool (November 2018).

Ten dimensions and 133 criteria were identified, both in relation to the outputs (Usability, Clinical Effectiveness, Security, Development, etc.) and the outcomes (such as weight loss, number of steps). Of those, 114 were included in the modified Delphi, in which 31 professionals participated. A set of 63 criteria were selected as candidates for being part of the tool. Criteria mainly belonged to Security (22%) and Usability dimensions (14%), followed by Quality (11%), and outcomes related to Activity (11%) and Physical status (11%). Once the co-creation session has been performed, the final tool will be developed.

Relevant criteria to evaluate the efficacy and safety of mHealth interventions in the management of overweight and obesity have been identified. Once the tool is developed it will be user tested and piloted on users of overweight and obesity management apps.