A methodologically rigorous appraisal of available evidence on safety outcomes of health care interventions is paramount for clinical decision-making. On the one hand, patients may be exposed to harm, if the severity or frequency of treatment adverse effects are overlooked; on the other hand, individuals with treatable disorders might not benefit from potentially effective treatments if adverse effects are overestimated. Additionally, efficacy or effectiveness of treatments needs always to be balanced against their tolerability: a potentially effective treatment associated with significant harm will very unlikely be considered as a first-line option.
Whilst randomised controlled trials (RCTs) are considered the less biased type of intervention studies, many treatment adverse effects are too uncommon or too long term to be detected in RCTs. Furthermore, some adverse effects may not have been considered when the RCT was designed or may not have been systematically and appropriately investigated during the RCT. As such, pharmacoepidemiological studies, i.e., observational studies where participants may be exposed to one or more treatments without a method of random assignment, need to be considered to assess the evidence on treatment safety profile.
Systematic reviews and meta-analyses are tools to, respectively, retrieve/appraise and statistically integrate available empirical evidence. Whereas guidance on conducting and presenting systematic reviews and meta-analyses of RCTs has been developed in detail (see, e.g., the authoritative Handbook of the Cochrane Collaboration: http://handbook.cochrane.org/), until recently no comprehensive recommendations were specifically available in relation to systematic reviews/meta-analyses of pharmacoepidemiological studies of safety outcomes. Indeed, the Cochrane Handbook provides one section only (No. 13) on non-randomised trials, which does not specifically focus on the appraisal of studies of safety outcomes, and one section only (No. 14) on appraisal of studies of adverse effects, which does not expressly target observational studies. Likewise, available guidance provided by other authoritative networks, such as the Centre for Reviews and Dissemination (http://www.york.ac.uk/crd/guidance/), does not focus specifically and in depth on safety outcomes from pharmacoepidemiological studies. In terms of reporting, the well-known Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (Liberati et al. Reference Liberati, Altman, Tetzlaff, Mulrow, Gotzsche, Ioannidis, Clarke, Devereaux, Kleijnen and Moher2009) focuses on health intervention studies (not necessarily observational ones). Another very influential document, the Meta-analysis of Observational Studies in Epidemiology (MOOSE) checklist (Stroup et al. Reference Stroup, Berlin, Morton, Olkin, Williamson, Rennie, Moher, Becker, Sipe and Thacker2000), is not specifically concerned with adverse effects assessment.
In December 2015, the European Network of Centres for Pharmacoepidemiology and Pharamacovigilance (ENCePP), a network coordinated by the European Medicines Agency (EMA), published their ‘Guidance on conducting systematic reviews and meta-analyses of completed comparative pharmacoepidemiological studies of safety outcomes’. This is a very welcome contribution to the field, filling a previous gap in the menu of available guidance/recommendations on conducting and reporting systematic reviews and meta-analyses.
Whilst some of the recommendations overlap with guidance for other types of meta-analyses, the ENCePP document provides a number of suggestions on aspects specifically related to systematic reviews/meta-analyses of pharmacoepidemiological studies of safety outcomes. I will highlight here those that, in my view, are the most relevant ones for each step of a systematic review/meta-analysis of pharmacoepidemiological studies. This overview will necessarily be selective due to space constraints.
Identification of studies
For a generic systematic review, empirical evidence shows that literature searches limited to electronic databases retrieve only approximately half of all relevant studies (Stroup et al. Reference Stroup, Berlin, Morton, Olkin, Williamson, Rennie, Moher, Becker, Sipe and Thacker2000). Therefore, it is recommended to search other sources, such as study registries, conference proceedings, grey (i.e., not controlled by commercial publishers) literature, and to contact experts in the field (Trespidi et al. Reference Trespidi, Barbui and Cipriani2011). In systematic reviews of RCTs, the search in sources other than electronic databases is made easier by the availability of specific repositories of randomised trials, such as clinicaltrials.gov, and by the requirement of ethical approval, which makes RCTs more easily traceable. Unfortunately, this is not always the case for pharmacoepidemiological studies. In this regard, the ENCePP suggests to considering additional sources, such as the Clinical Practice Research Database (CPRD) and the EU PAS Register, that are perhaps less well known to a number of researchers but that may turn out to be a precious source. Another challenge in the literature search process for systematic reviews of pharmacoepidemiological studies relates to the poor reliability of databases filters and indexing terms specifically tapping this type of studies. Indeed, whilst, for example, in Pubmed filters are available for RCTs, other study design labels are unlikely to capture all observational studies, in part owing to their heterogeneous nature. Therefore, the ENCePP urges caution in using such filters/indexing.
Data extraction
Whereas the process of data extraction would follow the same recommendations as for other systematic review (e.g., it should be conducted by two researchers independently), some of the extracted variables reflect the specificity of pharmacoepidemiological studies of safety outcomes. More specifically, the ENCePP highlights the relevance of reporting if the study was primarily aimed at assessing efficacy of intervention, safety endpoints, or if it was an aetiological study where medication use was only one of a range of factors considered. Additionally, the type of harm that the study could detect (related to the length of time participants were under observation), and the definition of safety outcomes are important variables to extract.
Study quality assessment
Many readers will be familiar with the GRADE system, including the Cochrane risk of bias tool, for RCTs, which is currently recommended as the standard for quality assessment of this type of studies (Barbui & Cipriani, Reference Barbui and Cipriani2011). There is less consensus on quality assessment tools for pharmacoepidemiological studies. Whilst the Cochrane collaboration has often used the Newcastle Ottawa Scale 7 (NOS) (Wells et al. Reference Wells, Shea, O'Connell, Peterson, Welch, Losos and Tugwell2010), the ENCePP points out that a drawback of this scale is the lack of an ‘Other Bias’ item and, of relevance in particular for pharmacoepidemiological studies, the inability to adapt the bias assessment to domains highly relevant for such studies, such as the so-called ‘confounding by indication’ (i.e., when a variable is a risk factor for a disease among non-exposed individuals and is associated with the exposure of interest in the population from which the cases derive). By contrast, more recent tools such as the RTI Item Bank, commissioned by the Agency for Healthcare Research and Quality (AHRQ) (Viswanathan & Berkman, Reference Viswanathan and Berkman2011), do take into account the ‘confounding by indication’. Arguably, the development and refinement of tools to appraise the quality of pharmacoepidemiological studies of safety outcomes will be one of the most relevant and expanding areas in the field.
Statistical plan
Two main issues arise when it comes to meta-analyse data from pharmacoepidemiological studies of safety outcomes. First, high inter-study variability is frequent due to, biases in treatment effect estimates arising from the lack of control over treatment allocation, absence of blinding, differences in length and completeness of follow-up. Whilst heterogeneity can be taken into account by means of random effect analytical approaches, the ENCePP points to the possibility of using Bayesian methods, especially when the number of meta-analysed studies is small. Furthermore, sensitivity analyses should always be planned to assess important sources of variability. Second, and relatedly, confounders need to be taken into account. Two approaches suggested by the ENCePP are: (1) to meta-analyse only the most credible adjusted results from the studies; and (2) to collect individual participant data with as many potential confounders as possible and to repeat the adjustments for the entire dataset.
The reader is invited to carefully examine the full text of the ENCePP guidance for further details on the recommended procedures for conducting systematic reviews and meta-analyses of pharmacoepidemiological studies of safety outcomes.
To conclude, although the ENCePP guidance should not be considered as a definitive document, since it will likely be refined in several iterations arising from researchers’ feedback, it is expected that it will be highly influential and useful for the field, with the aim to improve and standardise the conduction and reporting of systematic reviews and meta-analyses of pharmacoepidemiological studies of safety outcomes.
Acknowledgements
None.
Financial Support
This was not a funded study.
Conflict of Interest
The author declares co competing conflicts of interest.
