Reporting of harms in clinical trials of esketamine in depression: a systematic review

Abstract

While previous systematic reviews of trials evaluating conventional antidepressants highlighted inadequacies and inconsistencies in adverse event (AE) reporting, no evaluation is available on esketamine in resistant depression. The objective of this review was to assess quality of reporting AEs in all published clinical trials studying esketamine. It also aimed to compare the proportions of AEs reported in journal articles to those recorded in the ClinicalTrial.gov Registers. Clinical trials evaluating the efficacy and safety of esketamine in depression were searched using Medline and ClinicalTrials.gov. The quality of reporting harms was assessed using a 21-item checklist from the CONSORT Extension of Harms (1 point by item). The total quality score was graded into four categories: high (17–21), moderate (12–16), low (7–11) and very low (0–6). Ten clinical trials were included in the analysis. Nine trials were classified as ‘low quality’ with regard to safety, one trial was classified as ‘moderate quality’. Compared to AEs recorded in ClinicalTrials.gov, we found that 41.5% of serious AEs and 39% of non-serious AEs were not reported in the published articles. Among them, the majority were psychiatric events but also cardiovascular events and 94% concerned patients from esketamine groups. Quality of AEs reporting in published clinical trials of esketamine was poor and harms were reported less frequently in journal publications than in ClinicalTrial.gov Registers. The study suggests that an assessment of the benefits/risks balance of esketamine based on the results reported in trial publications is flawed due to the poor accuracy and completeness of harm data.

Introduction

Esketamine is a non-selective, non-competitive, antagonist of the N-methyl-D-aspartate receptor. Since March 2019, esketamine is approved by the Food and Drug Administration for adults with treatment-resistant major depressive disorder (US Food and Drug Administration, 2019). Esketamine has also been approved in November 2019 in Europe in the treatment of resistant depression (European Medical Agency, 2019). For both approvals, the efficacy criteria were based on three phase 3 clinical trials (TRANSFORM-1, TRANSFORM-2 and TRANSFORM-3), and one maintenance trial SUSTAIN-1. Among them, only TRANSFORM-2 was able to reach statistical significance. With the data currently available, the National Institute for Health and Care Excellence does not recommend the use of esketamine in depression (National Institute for Health and Care Excellence, 2020). The uncertainties regarding this drug are also shared by some authors (Cristea & Naudet, 2019; Horowitz & Moncrieff, 2021; Turner, 2019).

In 2022, the benefits/harms balance of esketamine is still debated in the scientific literature especially in the long term (Capuzzi et al., 2021; Kryst, Kawalec, & Pilc, 2020). Some studies suggest a lack of efficacy in treatment-resistant depression, and harm data are still limited (Gastaldon, Raschi, Kane, Barbui, & Schoretsanitis, 2021). Some information about harms of drugs could be obtained through randomized controlled trials (RCTs). Adverse events (AEs) reported in clinical trials play an important role in characterizing the harms/benefits balance. This is even more important when the drug is new and real-life studies are scarce. Thus, the way in which harms are reported in clinical trials becomes essential. The CONsolidated Standards Of Reporting Trials (CONSORT) statement is a tool that guides investigators to improve transparent and quality of publications (Schulz, Altman, Moher, & CONSORT Group, 2010). In 2004, this tool was adapted to harms with the CONSORT for harms checklist (Ioannidis et al., 2004). According this checklist, it was possible to quantify the quality of reporting AEs in clinical trials. Previous systematic review of clinical trials evaluating psychotropic drugs in depressive disorders highlighted inadequacies and inconsistencies in AE reporting (Meister et al., 2016).

Considering previous results on antidepressants and to support the increasing use of esketamine in resistant depression, there is a need to assess the quality of reporting AEs in clinical trials evaluating esketamine in depression. Therefore, the aim of this review was to assess how AEs were reported in all clinical trials published in scientific journals studying intranasal esketamine in depression. To improve reporting transparency, the 2007 Food and Drug Administration Amendments Act mandated the reporting of all clinical trial results in the publicly accessible ClinicalTrials.gov database (ClinicalTrials.gov, 2018). This is why our review also aims to compare reported AEs from these trials in journal articles to those recorded in the ClinicalTrial.gov Registers.

Methods

Protocol and registration

The systematic review protocol was written in agreement with the Preferred Reporting Items for Systematic Review and Meta-analysis Protocols (PRISMA-P) (Zorzela et al., 2016). The present review was reported in accordance with the PRISMA statement. The protocol was registered with PROSPERO with the number CRD42022329991 (https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=329991).

Eligibility criteria and study selection

Since the objective was to study the available data that were easily accessible, we only looked for published data. To be eligible, clinical trials had to focus on a population of human subjects suffering from depressive disorders, treated with intranasal esketamine. In order to have a significant exposure close to real life use, exposure to esketamine had to be greater than 7 days. The presence of a comparator group was not necessary, if present, any comparator could be accepted. Clinical trials should study the efficacy and/or safety of esketamine in depression. The search was conducted via the PubMed database (Medline) and the ClinicalTrial.gov Registers. For the PubMed database, the search was performed with the function [esketamine AND (depression OR major depressive disorder OR depressive disorder)]. All references were then retrieved. For the ClinicalTrial.gov Registers database, the search was performed using the following criteria: condition disease [depression], other term [esketamine], status [terminated] and study results [with results]. After removing duplicates from both searches, we did a first selection on the title. Then, a selection on the abstracts was performed to check which publications were eligible. For these first two steps, all titles or abstracts mentioning an indication other than depressive disorders, or a methodology other than a clinical trial have been excluded. Finally, a last selection on the full text was made to determine which studies would be included. For this purpose, the trials should satisfy the eligibility criteria (including indication and route of administration) mentioned above.

Data collection process

All data have been manually extracted from the ClinicalTrial.gov Registers and from published articles found in Medline and downloaded from the respective journal sites. For each trial, we manually extracted:

• – General characteristics: date of publication, name of trial, clinical development phase, number of centers, number of arms, blinding procedures, number of participants randomized, the primary outcome, inclusion and exclusion criteria's, funding, the first author's affiliation

• – Information on AEs, if available for both sources and for each arm: all reported AEs have been classified according to the MEDdra classification. Number of patents experiencing at least one AE, number of patients discontinuing trial due to AEs, threshold for reporting AE. All available evidence on how to track AEs in published articles was also to be reviewed.

Risk of bias in individual studies

The risk of bias was assessed using the Cochrane risk-of-bias tool for randomized trials second version (RoB-2) for each included study (Sterne et al., 2019). This tool is a standardized method for assessing potential bias in reports of randomized interventions, consisting of a fixed set domain of bias due to randomization process, deviations from intended interventions, missing outcome data, measurement of the outcomes, and selection of reported results. A proposed judgment about the risk of bias arising from each domain was generated by an algorithm, where judgment can be ‘low’ or ‘high’ risk of bias or can express ‘some concerns’. For non-randomized studies, the risk of bias was assessed using the Risk of Bias in Non-randomized Studies of Interventions (ROBINS-I) tool. Similarly, ROBINS-I, which is structured into a fixed set of domains of bias, includes signaling questions that inform the risk of bias judgments, and on the basis of answers to the questions, judgments for each bias domain, and the overall risk of bias can be classified as ‘low’, ‘moderate’, ‘serious’, or ‘critical’ (Sterne et al., 2016).

Quality of reporting of harm data

The harm reporting data from all included trials was independently assessed by two authors (T. T. L. and F. M.) using CONSORT Extension of Harms 2004, a 10-point checklist (Ioannidis et al., 2004). In case of disagreement, a third author (A. J.) was consulted. The 10 items of the CONSORT Extension for Harms checklist being quite broad and general, to allow a more precise and reproducible assessment we used the 21-item checklist (Table 1), based on the previously mentioned checklist already used by some authors (Mazhar et al., 2020; Yuniar et al., 2022). Thus, each item of the CONSORT checklist is broken down into several more specific criteria. Each item in the list was scored individually and weighted with equal importance, according to the item CONSORT recommendations. Each carried a score of 1 if it was adequately reported or 0 if it was inadequately reported or not reported at all. The total score was calculated by adding up all the individual scores, called the Total Harm Reporting Score (THRS). The THRS was then ranked: high quality: 17–21; moderate quality: 12–16; low quality: 7–11; very low quality: 0–6 (Kow, Aldeyab, & Hasan, 2021).

Table 1. Quality of reporting criteria [Consolidated Standards of Reporting Trials (CONSORT) extension for harm] and compliance of trials for each item

CONSORT, CONsolidated Standards Of Reporting Trials; AE, adverse event.

Adverse events analysis

Descriptive analyses of AEs are presented as proportions for qualitative variables. To determine whether the AEs reported in published articles are similar to those recorded in ClinicalTrial.gov Registers, serious AEs and ‘non serious’ AEs were analyzed separately. The comparison was done trial by trial by analyzing separately AEs that occurred in the induction phase of esketamine (from week 1 to week 4), those that occurred during the maintenance phase (from week 5) and those that occurred at discontinuation of the drug (follow-up). To calculate the percentage of missing AEs, we calculated the total number of AEs reported with published articles and ClinicalTrial.gov. For this, we added to the AEs reported in both ClinicalTrial and published articles the AEs not reported in ClinicalTrial.gov (and reported in published articles) and those not reported in published articles (and present in ClinicalTrial.gov). There were no changes to the protocol initially filed.

Results

Study selection

From our search and selection strategy, 436 references were identified. After screening, 10 trials (numbered from ID1 to ID10) were eligible and were included in this review with a total of 2597 subjects (Fig. 1) (Canuso et al., 2018; Daly et al., 2018, 2019; Fedgchin et al., 2019; Fu et al., 2020; Ionescu et al., 2021; Ochs-Ross et al., 2020; Popova et al., 2019; Takahashi et al., 2021; Wajs et al., 2020).

Figure 1. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram of the study selection process.

Study characteristics

A summary of characteristics of the included studies is provided in online Supplementary Table S1. Trials were published between 2019 and 2021. Among the 10 studies, nine were double-blind parallel-group placebo-controlled RCTs and one (study ID5, NCT02497287) was a non-randomized open-label clinical trial. This study was the only one specially designed for long-term safety purpose. All RCTs had two arms: esketamine + antidepressant (ESK) or placebo + antidepressant (PBO). The majority of studies (7/10) were phase 3 trials. Nine trials included participants aged 18–64, only one included subjects aged over 64 (study ID3, NCT02422186). The 10 clinical trials selected were carried out in several phases. For the 10 trials, the lead author's affiliation was the same, namely by the pharmaceutical company that markets esketamine.

Risk of bias

The risk of bias was analyzed using the Rob2 tool for nine trials (online Supplementary Fig. S1). The ID5 trial could not be assessed using the RoB-2 tool or the ROBIN-I tool because it was not randomized and did not have a comparator arm. All nine evaluated trials were categorized as trials with ‘some concerns’. For each of them, domain 2: ‘deviations from the intended interventions’ was responsible for the ‘some concerns’ outcome. Indeed, it was considered that it was possible for the participants and the investigators to know the assigned intervention during the study because of AEs of esketamine (especially dissociative disorders). For trials ID6 to ID9 (NCT03039192, NCT03097133, NCT02133001, NCT01998958), domain 4: ‘measurement of the outcome’ was considered with ‘some concerns’. Similarly, evaluators of the MADRS score could know the assigned intervention due to AEs. In the other trials, the MADRS assessments were performed via telephone by blinded independent raters.

Adherence to CONSORT recommendations

The median THRS was 10 with a minimum of 9 and a maximum of 14 (Table 2). According to the classification stated above, all trials except ID5 (NCT02497287) were qualified as ‘low quality’, and the ID5 trial was qualified as ‘moderate quality’. The compliance rates for each item of the CONSORT for harms scale are reported in Table 1. Several items were missing in each of the 10 clinical trials: 3a (definition of AEs), 4a (mode of data collection), 4c (description of how AEs were attributed to trial drugs), 4d (description of the plan for monitoring for harms and rules for stopping the trial because of harms), 5b (description of approach for the handling of recurrent AEs) and 10a (provided a balanced view that puts benefits and harms into perspective in discussion). Item 8b (severity and grading of AEs) was only met in trial ID10 (NCT02918318). Indeed, for item 8b, all the other clinical trials simply specified ‘Most adverse events were of mild or moderate in severity’ which was considered too imprecise. Item 8c (provided both number of AEs and number of patients with AEs) was more inconsistent across trials.

Table 2. Total Harm Reporting Score results for CONSORT for harms items

THRS, Total Harm Reporting Score.

Adverse events

In ClinicalTrial.gov Registers, we found 9464 AEs recorded (v. 5859 reported in published articles) including 179 ‘serious’ AEs (v. 130 reported in published articles). The threshold for reporting AEs was 5% in all records available on ClinicalTrial.gov Registers, while it was sometimes 10% in published articles for studies ID5 (NCT02497287), ID8 (NCT02133001) and ID9 (NCT01998958). When mentioned, the number of patients with at least one AE was generally different between ClinicalTrial.gov Registers and published articles, as well as AEs leading to discontinuation. In 7/10 of the trials, AEs and ‘serious’ AEs in the follow-up phases were not described in the published articles even though the follow-up phases were reported in the part describing withdrawal syndromes. Conversely, for the ID6 (NCT03039192) and ID7 (NCT03097133) trials, ClinicalTrial.gov Registers did not mention AEs of the follow-up phase (Table 3). The ID9 (NCT01998958) trial was designed with two patient panels; only the information on the first panel was available in the published article. About the proportion of unreported AEs, 39.0% of the ‘non-serious’ AEs were not mentioned in the published articles and 41.5% of ‘serious’ AEs were not mentioned in the published articles. Conversely, 1.4% of ‘non-serious’ AEs and 20.8% of ‘serious’ AEs were not recorded in the ClinicalTrial.gov Register (Table 4).

Table 3. Summary of adverse events reported in published articles and ClinicalTrial.gov Registers for each clinical trial by phase

PA, published articles; CTR, ClinicalTrial.gov Register.

Table 4. Numbers of adverse events not reported in ClinicalTrial.gov Registers and published articles

CTR, ClinicalTrial.gov Registers; PA, published articles; AE, adverse events.

^a Number of adverse events (and percentage) not mentioned in the source studied out of the total number of adverse events.

^b Number of serious adverse events (and percentage) not mentioned in the source studied out of the total number of adverse events.

^c The synthesis is calculated by adding to the AEs reported in both ClinicalTrial and in the published articles the AEs not reported in ClinicalTrial.gov (and present in the published articles) and those not reported in the published articles (and reported in ClinicalTrial.gov).

The majority of ‘serious’ AEs not mentioned in published articles (88/94) concerned patients from esketamine groups (Table 5 and Fig. 2). Most (71/94) of these ‘serious’ AEs occurred in the phases of direct exposure to esketamine or placebo (induction, optimization, or maintenance phases). Among the ‘serious’ AEs not mentioned in the published articles and found in the patients of the esketamine group, we mainly found psychiatric but also cardiovascular (cerebral hemorrhage, hypertensive crisis), kidney, and urinary (nephrolithiasis, tubulointerstitial nephritis) symptoms. Among these psychiatric effects not found in published articles, two suicide attempts and one completed suicide were identified.

Figure 2. Summary of ‘serious’ AEs not reported in published articles according Meddra classification by treatment group.

Table 5. Summary of ‘serious’ AEs not reported in published articles according Meddra classification by phases and treatment group in published articles

ESK, esketamine + antidepressant; PBO, placebo + antidepressant; SOC, System Organ Classes; LLT, Lowest Level Term.

Discussion

This systematic review was conducted to assess and summarize the quality of harms reporting in clinical trials evaluating efficacy and safety of intranasal esketamine in depressive disorders. All published clinical trials of esketamine in the treatment of resistant depression show that the quality of AEs reported in clinical trials evaluating esketamine was low. Compared to AEs recorded in ClinicalTrials.gov, we found also discrepancies in the number of AEs reported in journal publications. More than two-fifths of serious events and one-fifth of non-serious events were not reported in clinical trials published articles. Among the ‘serious’ AEs of esketamine not reported in the published articles, we mainly found psychiatric, cardiovascular, kidney and urinary symptoms. These events have already been the subject of questioning (Gastaldon et al., 2021; Horowitz & Moncrieff, 2021). These questions are all the more important with regard to the risk of suicide, in particular in view of the use of intravenous ketamine in reduction of suicidal ideation in treatment-resistant depression (Phillips et al., 2020).

Even though nine out of 10 trials had the primary objective of demonstrating esketamine efficacy, it can be expected that articles published in scientific journals would include a minimum of information to assess the harms/benefits balance. It should be noted that six of the 10 trials (ID1, 2, 3, 8, 9, 10) included terms ‘efficacy and safety’ in the title of the published article and were classified as ‘low quality’ in our review regarding safety. In a systematic review that studied the application of the CONSORT for harms recommendations in clinical trials studying conventional antidepressants (Meister et al., 2016), the authors used the 10-item CONSORT tool and analyzed 16 clinical trials. They showed that the mean number of CONSORT items that were fulfilled was 4.42/10 for a median equal to 5. In our case, the mean was 10.5/21 for a median of 10. These results show that the quality of AEs reporting with esketamine is close to that found by these authors. The authors concluded that ‘there is a strong need to improve the current practice of assessing, analyzing, and reporting AEs’. The CONSORT for harms recommendations on AEs was issued to help to identify the key information that needs to be included in a publication to properly report issues related to harms (Ioannidis et al., 2004). Here, we show that the data available on esketamine in published journal articles were often of poor quality with regard to harm reporting. This agrees with what some authors have already mentioned in general about clinical trials (Ioannidis & Lau, 2001; Papanikolaou, Churchill, Wahlbeck, & Ioannidis, 2004). This is also true for newer drugs, or at least for their use in new indications. For example, poor quality reporting of harms was observed during the coronavirus pandemic with several drugs (Kow et al., 2021; Mazhar et al., 2020). Importantly, none of the articles describe how the AEs were attributed to the drug. Some articles stated that they only reported AEs for which a link to the drug has been made, but they did not mention how this causality assessment was achieved.

The second part of the study on the comparison of the information in the published articles and the ClinicalTrial.gov Registers showed that a large number of AEs (39% of ‘non-serious’ AEs and 41.5% of ‘serious’ AEs) were not reported, mainly in the published articles. In general, this difference could be explained by the fact that the data on AEs in the published articles were referring to only a single phase (most often induction) and did not provide any further information on AEs found in the other phases of the trial (Daly et al., 2018, 2019; Fedgchin et al., 2019; Ochs-Ross et al., 2020; Popova et al., 2019; Takahashi et al., 2021). It should be noted that in some papers the authors gave efficacy information on the optimization (ID4), or open-label (ID9) phases without giving data on AEs in these same periods. Similarly, the ID1-5 trials provided data on withdrawal syndromes during the follow-up phase without mentioning AEs found during this same phase. However, we note that not all AEs that occur during the full duration of the trial are reported in the published articles. According to the CONSORT for harms scale, the ID5 study had the highest THRS. This is encouraging as the main objective of this study was to investigate safety data after longer exposure to the drug.

The frequency threshold for reporting AEs was generally 5% while in three trials (ID5, 8, 9), this threshold was 10%. This may have contributed to the difference observed in terms of the number of reported AEs. However, we can wonder about the choice of this threshold. Indeed, it takes 78 patients in the ID5 induction phase trials to develop the AE before it will be mentioned in the article. Moreover, the use of such a threshold is problematic insofar as AE can be arbitrarily classified by the investigator in one category or another. Only one of the 10 included trials met the ‘Described the methods for presenting and/or analyzing AEs’ item of the CONSORT for harms scale. This limits the sharing of information with regard to rare AEs which are already poorly detected normally in clinical trials. We included all the published clinical trials, all conducted by the same pharmaceutical company. The way of reporting the data relating to harms was therefore very close between the different trials. Regarding limits, note that our search was based on only two databases Medline and ClinicalTrial.gov and did not include any unpublished work.

Despite the existence of CONSORT recommendations, the systematic review suggests that quality of AEs reporting in published clinical trials of esketamine was poor. An assessment of the benefits/risks balance of esketamine based on the results reported in trial publications is flawed due to the poor accuracy and completeness of harm data. Added to the lack of transparency regarding unreported AEs in published articles, this raises questions about the speed of esketamine marketing approval. In order to improve this finding, authors and editors should use the CONSORT extension for harms scale more frequently. Finally, these discrepancies results confirm that post-marketing studies of AEs in real life are needed (Gastaldon et al., 2021).