Sharing, harmonizing, and analyzing participant-level data is of central importance in the rapid research response to emerging pathogens. Individual participant data meta-analyses (IPD-MAs), which synthesize participant-level data from related primary studies, have several advantages over pooling study-level effect estimates in a traditional meta-analysis. IPD-MAs enable researchers to more effectively separate spurious heterogeneity related to differences in measurement from clinically relevant heterogeneity from differences in underlying risk or distribution of factors that modify disease progression. This tutorial describes the steps needed to conduct an IPD-MA of an emerging pathogen and how IPD-MAs of emerging pathogens differ from those of well-studied exposures and outcomes. We discuss key statistical issues, including participant- and study-level missingness and complex measurement error, and present recommendations. We review how IPD-MAs conducted during the COVID-19 response addressed these statistical challenges when harmonizing and analyzing participant-level data related to an emerging pathogen. The guidance presented here is based on lessons learned in our conduct of IPD-MAs in the research response to emerging pathogens, including Zika virus and COVID-19.

Critical interpretive synthesis was introduced in 2006 to address various shortcomings of systematic reviews such as their limitations in synthesizing heterogeneous data, integrating diverse study types, and generating theoretical insights. This review sought to outline the methodological process of conducting critical interpretive syntheses by identifying the methods currently in use, mapping the processes that have been used to date, and highlighting directions for further research. To achieve this, a scoping review of critical interpretive syntheses published between 2006 and 2023 was conducted. Initial searches identified 1628 publications and after removal of duplicates and exclusions, 212 reviews were included in the study. Most reviews focused on health-related subjects. Authors chose to utilize the method due to its iterative, inductive, and recursive nature. Both question-based and topic-based reviews were conducted. Literature searches relied on electronic databases and reference chaining. Mapping to the original six-phase model showed most variability in use of sampling and quality assessment phases, which were each done in 50.7% of reviews. Data extraction utilized a data extraction table. Synthesis involved constant comparison, critique, and consolidation of themes into constructs, and a synthesizing argument. Refining critical interpretive synthesis methodology and its best practices are important for optimizing the utility and impact and ensuring findings are relevant and actionable for informing policy, practice, and future research.

Large language models have shown promise for automating data extraction (DE) in systematic reviews (SRs), but most existing approaches require manual interaction. We developed an open-source system using GPT-4o to automatically extract data with no human intervention during the extraction process. We developed the system on a dataset of 290 randomized controlled trials (RCTs) from a published SR about cognitive behavioral therapy for insomnia. We evaluated the system on two other datasets: 5 RCTs from an updated search for the same review and 10 RCTs used in a separate published study that had also evaluated automated DE. We developed the best approach across all variables in the development dataset using GPT-4o. The performance in the updated-search dataset using o3 was 74.9% sensitivity, 76.7% specificity, 75.7 precision, 93.5% variable detection comprehensiveness, and 75.3% accuracy. In both datasets, accuracy was higher for string variables (e.g., country, study design, drug names, and outcome definitions) compared with numeric variables. In the third external validation dataset, GPT-4o showed a lower performance with a mean accuracy of 84.4% compared with the previous study. However, by adjusting our DE method, while maintaining the same prompting technique, we achieved a mean accuracy of 96.3%, which was comparable to the previous manual extraction study. Our system shows potential for assisting the DE of string variables alongside a human reviewer. However, it cannot yet replace humans for numeric DE. Further evaluation across diverse review contexts is needed to establish broader applicability.

AMSTAR-2 (A Measurement Tool to Assess Systematic Reviews, version 2) and ROBIS are tools used to assess the methodological quality and the risk of bias in a systematic review (SR). We applied AMSTAR-2 and ROBIS to a sample of 200 published SRs. We investigated the overlap in their methodological constructs, responses by item, and overall, percentage agreement, direction of effect, and timing of assessments. AMSTAR-2 contains 16 items and ROBIS 24 items. Three items in AMSTAR-2 and nine in ROBIS did not overlap in construct. Of the 200 SRs, 73% were low or critically low quality using AMSTAR-2, and 81% had a high risk of bias using ROBIS. The median time to complete AMSTAR-2 and ROBIS was 51 and 64 minutes, respectively. When assessment times were calibrated to the number of items in each tool, each item took an average of 3.2 minutes per item for AMSTAR-2 compared to 2.7 minutes for ROBIS. Nine percent of SRs had opposing ratings (i.e., AMSTAR-2 was high quality while ROBIS was high risk). In both tools, three-quarters of items showed more than 70% agreement between raters after extensive training and piloting. AMSTAR-2 and ROBIS provide complementary rather than interchangeable assessments of systematic reviews. AMSTAR-2 may be preferable when efficiency is prioritized and methodological rigour is the focus, whereas ROBIS offers a deeper examination of potential biases and external validity. Given the widespread reliance on systematic reviews for policy and practice, selecting the appropriate appraisal tool remains crucial. Future research should explore strategies to integrate the strengths of both instruments while minimizing the burden on assessors.

A growing number of oncology treatments, such as bevacizumab, are used across multiple indications. However, in health technology assessment (HTA), their clinical and cost-effectiveness are typically appraised within a single target indication. This approach excludes a broader evidence base across other indications. To address this, we explored multi-indication meta-analysis methods that share evidence across indications.

We conducted a simulation study to evaluate alternative multi-indication synthesis models. This included univariate (mixture and non-mixture) methods synthesizing overall survival (OS) data and bivariate surrogacy models jointly modeling treatment effects on progression-free survival (PFS) and OS, pooling surrogacy parameters across indications. Simulated datasets were generated using a multistate disease progression model under various scenarios, including different levels of heterogeneity within and between indications, outlier indications, and varying data on OS for the target indication. We evaluated the performance of the synthesis models applied to the simulated datasets in terms of their ability to predict OS in a target indication.

The results showed univariate multi-indication methods could reduce uncertainty without increasing bias, particularly when OS data were available in the target indication. Compared with univariate methods, mixture models did not significantly improve performance and are not recommended for HTA. In scenarios where OS data in the target indication is absent and there are also outlier indications, bivariate surrogacy models showed promise in correcting bias relative to univariate models, though further research under realistic conditions is needed.

Multi-indication methods are more complex than traditional approaches but can potentially reduce uncertainty in HTA decisions.

The contrast-based model (CBM) is the most popular network meta-analysis (NMA) method, although alternative approaches, e.g., the baseline model (BM), have been proposed but seldom used. This article aims to illuminate the difference between the CBM and BM and explores when they produce different results. These models differ in key assumptions: The CBM assumes treatment contrasts are exchangeable across trials and models the reference (baseline) treatment’s outcome levels as fixed effects, while the BM further assumes that the baseline treatment’s outcome levels are exchangeable across trials and treats them as random effects. We show algebraically and graphically that the difference between the CBM and BM is analogous to the difference between the two analyses in a statistical conundrum called Lord’s Paradox, in which the t-test and analysis of covariance (ANCOVA) yield conflicting conclusions about the group difference in weight gain. We show that this conflict arises because the t-test compares the observed weight change, whereas ANCOVA compares an adjusted weight change. In NMA, analogously, the CBM compares observed treatment contrasts, while the BM compares adjusted treatment contrasts. We demonstrate how the difference in modeling baseline effects can cause the CBM and BM to give different results. The analogy of Lord’s Paradox provides insights into the different assumptions of the CBM and BM regarding the relationship between baseline effects and treatment contrasts. When these two models produce substantially different results, it may indicate a violation of the transitivity assumption. Therefore, we should be cautious in interpreting the results from either model.

This study analyzes 1,000 meta-analyses drawn from 10 disciplines—including medicine, psychology, education, biology, and economics—to document and compare methodological practices across fields. We find large differences in the size of meta-analyses, the number of effect sizes per study, and the types of effect sizes used. Disciplines also vary in their use of unpublished studies, the frequency and type of tests for publication bias, and whether they attempt to correct for it. Notably, many meta-analyses include multiple effect sizes from the same study, yet fail to account for statistical dependence in their analyses. We document the limited use of advanced methods—such as multilevel models and cluster-adjusted standard errors—that can accommodate dependent data structures. Correlations are frequently used as effect sizes in some disciplines, yet researchers often fail to address the methodological issues this introduces, including biased weighting and misleading tests for publication bias. We also find that meta-regression is underutilized, even when sample sizes are large enough to support it. This work serves as a resource for researchers conducting their first meta-analyses, as a benchmark for researchers designing simulation experiments, and as a reference for applied meta-analysts aiming to improve their methodological practices.

In many areas of oncology, cancer drugs are now associated with long-term survivorship and mixture cure models (MCM) are increasingly being used for survival analysis. The objective of this article was to propose a methodology for conducting network meta-analysis (NMA) of MCM. This method was illustrated through a case study evaluating recurrence-free survival (RFS) with adjuvant therapy for stage III/IV resected melanoma. For the case study, the MCM NMA was conducted by: (1) fitting MCMs to each trial included within the network of evidence; and (2) incorporating the parameters of the MCMs into a multivariate NMA. Outputs included relative effect estimates for the MCM NMA as well as absolute estimates of survival (RFS), modeled within the Bayesian multivariate NMA, by incorporating absolute baseline effects of the reference treatment. The case study was intended for illustrative purposes of the MCM NMA methodology and is not meant for clinical interpretation. The case study demonstrated the feasibility of conducting an MCM NMA and highlighted key issues and considerations when conducting such analyses, including plausibility of cure, maturity of data, process for model selection, and the presentation and interpretation of results. MCM NMA provides a method of comparative survival that acknowledges the benefit newer treatments may confer on a subset of patients, resulting in long-term survival and reflection of this survival in extrapolation. In the future, this method may provide an additional metric to compare treatments that is of value to patients.

The ICH E9(R1) addendum provides guidelines on accounting for intercurrent events in clinical trials using the estimands framework. However, there has been limited attention to the estimands framework for meta-analysis. Using treatment switching, a well-known intercurrent event that occurs frequently in oncology, we conducted a simulation study to explore the bias introduced by pooling together estimates targeting different estimands in a meta-analysis of randomized clinical trials (RCTs) that allowed treatment switching. We simulated overall survival data of a collection of RCTs that allowed patients in the control group to switch to the intervention treatment after disease progression under fixed effects and random effects models. For each RCT, we calculated effect estimates for a treatment policy estimand that ignored treatment switching, and a hypothetical estimand that accounted for treatment switching either by fitting rank-preserving structural failure time models or by censoring switchers. Then, we performed random effects and fixed effects meta-analyses to pool together RCT effect estimates while varying the proportions of trials providing treatment policy and hypothetical effect estimates. We compared the results of meta-analyses that pooled different types of effect estimates with those that pooled only treatment policy or hypothetical estimates. We found that pooling estimates targeting different estimands results in pooled estimators that do not target any estimand of interest, and that pooling estimates of varying estimands can generate misleading results, even under a random effects model. Adopting the estimands framework for meta-analysis may improve alignment between meta-analytic results and the clinical research question of interest.

To understand a treatment’s potential impact at the individual level, it is crucial to explore whether the effect differs across patient subgroups and covariate values. Meta-analysis provides an important tool for detecting treatment–covariate interactions, as it can improve power compared to a single study. However, aggregation bias can occur when estimating individual-level treatment–covariate interactions in meta-analysis, due to trial-level confounding. This refers to when the association between the covariate and treatment effect across trials (at the aggregate level) differs from that observed within trials (at the individual level). It is, thus, recommended that heterogeneity in the treatment effect at the individual level should be disentangled from that at the trial level, ideally using an individual participant data (IPD) meta-analysis. Here, we explain this issue and provide new intuition about how trial-level confounding is impacted by differences in within-trial distributions of covariates and how this corresponds to asymmetry in subgroup-specific funnel plots in the case of categorical covariates. We then propose a sensitivity analysis to assess the robustness of interaction estimates to potential trial-level confounding. We illustrate these concepts using simulated and real data from an IPD meta-analysis of trials conducted on the TICO/ACTIV-3 platform, which assessed passive immunotherapy treatments for inpatients with COVID-19.

Umbrella reviews (URs) synthesize findings from multiple systematic reviews on a specific topic. Methodological approaches for analyzing and presenting UR results vary, and reviewers often adapt methods to align with research objectives. This study examined the characteristics of analysis and presentation methods used in healthcare-related URs. A systematic PubMed search identified URs published between 2023 and 2024. Inclusion criteria focused on healthcare URs using systematic reviews as the unit of analysis. A random sample of 100 eligible URs was included. A customized, piloted data extraction form was used to collect bibliographic, conduct, and reporting data independently. Descriptive analysis and narrative synthesis summarized findings. The most common terminology for eligible studies was “umbrella reviews” (65%) or “overviews” (30%). Question frameworks included PICO (43%) and PICOS (14%), with quantitative systematic reviews included in most URs (98%), and 68% including randomized controlled trials. The most frequent methodological guidance source was Cochrane (32%). Data analysis commonly used narrative synthesis and meta-analysis, with Stata, RevMan, and GRADEPro GDT employed for presentation. Information about study overlap and certainty assessment was rarely reported.Variation exists in how data are analyzed and presented in URs, with key elements often omitted. These findings highlight the need for clearer methodological guidance to enhance consistency and reporting in future URs.

Existing guidelines on overviews of reviews and umbrella reviews recommend an assessment of the certainty of evidence, but provide limited guidance on ‘how to’ apply the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) to such a complex evidence synthesis. We share our experience of developing a ‘general principles’ approach to applying GRADE to a complex overview of reviews. The approach was developed in an iterative and exploratory manner during the planning and conduct of an overview of reviews of a novel molecular imaging technique for the staging of prostate cancer, involving a formal review by a group of 11 methodologists/health services researchers. This approach was developed during the evidence synthesis process, piloted, and then applied to our ongoing overview of reviews. A ‘general principles’ approach of applying the domains of GRADE to an overview of reviews and arriving at an overall summary judgement for each outcome is presented. Our approach details additional factors to consider, including addressing both the primary study risk of bias as assessed by the included reviews and the risk of bias of the systematic reviews themselves, as well as the statistical heterogeneity observed in meta-analyses conducted within the included reviews. Our approach distilled key principles from the relevant GRADE guidelines and allowed us to apply GRADE to a complex body of evidence in a consistent and transparent way. The approach taken and the methods used to develop our approach may inform researchers working on overviews of reviews, umbrella reviews, or future methodological guidelines.