Systematic syntheses of individual trials have been described as the ‘gold standard’ for the evaluation of interventions (Sackett et al. Reference Sackett, Rosenberg, Gray, Haynes and Richardson1996, p. 72). As pointed out by de Jonghe et al., systematic reviews and meta-analyses play an increasing role in the decision making of clinicians, researchers and policy makers. However, meta-analyses are, of course, not immune from bias. In their letter, de Jonghe et al. criticize our recent network meta-analysis of psychological interventions for post-traumatic stress disorder (PTSD) (Gerger et al. Reference Gerger, Munder, Gemperli, Nüesch, Trelle, Jüni and Barth2014 b) for severe methodological shortcomings and question the relevance of our study. Many of the issues raised by de Jonghe et al. have already been considered in our paper. However, we would like to use the opportunity of this rejoinder to further clarify some issues.
de Jonghe et al. seem to be unsatisfied with our finding of equivalent effectiveness of specific psychological interventions. They argue that we did not identify superiority of any intervention ‘due to the considerable between-trial heterogeneity’, which they attribute to our inclusion of heterogeneous samples. The first part of this statement lacks scientific evidence, however. As described in the Introduction of our paper, the large majority of meta-analyses in the field of PTSD interventions conclude equivalent effectiveness of specific interventions (e.g. Bisson & Andrew, Reference Bisson and Andrew2007; Watts et al. Reference Watts, Schnurr, Mayo, Young-Xu, Weeks and Friedman2013) and none of the interventions has consistently been shown to outperform the others; not even in meta-analyses with less between-trial heterogeneity (e.g. Benish et al. Reference Benish, Imel and Wampold2008). However, as stated in our Limitations we admit that we did not control for possibly moderating effects of clinical patient characteristics, which have previously been shown to affect relative effect size estimates (Gerger et al. Reference Gerger, Munder and Barth2014 a). We have, however, conducted a moderator analysis including the status of a full PTSD diagnosis (versus subclinical PTSD symptoms) to explain heterogeneity, which de Jonghe and colleagues may have overseen in our paper.
A further point of critique is that we distinguished trials with small to moderate samples from trials with larger samples. We elaborated extensively on the rationale for the cut-offs chosen in our analyses in the Method. We are therefore not clear about the actual critique here. Our cut-offs conform to those proposed by Schnurr (Reference Schnurr2007), which also rely on power considerations. Given the vast literature on the risk of bias that is typically associated with small samples (Egger et al. Reference Egger, Smith, Schneider and Minder1997; Sterne et al. Reference Sterne, Gavaghan and Egger2000; Cuijpers et al. Reference Cuijpers, Smit, Bohlmeijer, Hollon and Andersson2010a , Reference Cuijpers, van Straten, Bohlmeijer, Hollon and Andersson b ; Nüesch et al. Reference Nüesch, Trelle, Reichenbach, Rutjes, Tschannen, Altman, Egger and Jüni2010; Barth et al. Reference Barth, Munder, Gerger, Nüesch, Trelle, Znoj, Jüni and Cuijpers2013; Watts et al. Reference Watts, Schnurr, Mayo, Young-Xu, Weeks and Friedman2013), we do not believe that the authors aimed at fundamentally questioning the relevance of sample size as a moderator variable.
Furthermore, our conclusion of eye movement desensitization and reprocessing (EMDR) as ‘promising’, which is not negative in principle, seems to contradict de Jonghe et al.'s expectations. The authors argue that, in the presence of more than 20 trials on EMDR, our conclusion ‘lacks scientific merit’. However, from our point of view and based on the extensive empirical literature on small sample bias, we feel very confident in repeating the conclusion regarding the lack of robust evidence for EMDR. We were unable to identify a single trial on the efficacy of EMDR that was adequately sized to detect relative intervention effects of moderate to small size. We therefore strongly argue for the need for collaborative research projects (such as the Social Phobia Psychotherapy Network by Leichsenring et al. Reference Leichsenring, Hoyer, Beutel, Herpertz, Hiller, Irle, Joraschky, Konig, de Liz, Nolting, Pohlmann, Salzer, Schauenburg, Stangier, Strauss, Subic-Wrana, Vormfelde, Weniger, Willutzki, Wiltink and Leibing2009) that aim at maximizing the number of patients included in a comparative trial and at minimizing the potential for bias from researchers’ preferences (the so-called allegiance bias; see Munder et al. Reference Munder, Gerger, Trelle and Barth2011, Reference Munder, Flückiger, Gerger, Wampold and Barth2012) at the same time. Our evaluation of the evidence for EMDR also mirrors the appraisal of the Society of Clinical Psychology (APA Division 12) that summarizes the evidence for EMDR as ‘strong research support/controversial’. We admit, however, that more recent publications may reveal additional evidence from larger trials on the effectiveness of EMDR.
Referring to the last statement, de Jonghe and colleagues criticize our study for ending our search in 2010 and thereby missing ‘a number of studies’. We agree that a more recent search of trials would be preferable. However, we learned from two previous updates of our meta-analysis that effect estimates and their precision changed only slightly. The main results remained constant across the updates. The reliability of the major findings regarding the equivalent effectiveness of individual types of intervention may also be regarded as confirmed by several previous meta-analyses on PTSD interventions (e.g. Bradley et al. Reference Bradley, Greene, Russ, Dutra and Westen2005; Bisson & Andrew, Reference Bisson and Andrew2007) and also by the most recent meta-analysis by Watts et al. (Reference Watts, Schnurr, Mayo, Young-Xu, Weeks and Friedman2013). The conclusions of these meta-analyses are very similar to ours. We summarized these results in the Introduction of our paper.
We agree with de Jonghe and colleagues that our meta-analysis does not provide radically new insights with regard to the classic horse race question. However, the major advantage of our meta-analysis compared to other meta-analyses in the field of PTSD interventions is that we applied a novel approach of analysis, namely network meta-analysis. By summarizing all available evidence from studies with multiple comparisons and diverse control groups in a single analysis, network meta-analysis allowed us to address the question of relative intervention effects more elegantly and with more statistical power. Thus, our network meta-analysis allowed us to go beyond the classic horse race question and explore sources of heterogeneity and alternative ways of clustering interventions, and in particular allowed us to conduct analyses with a relatively small number of trials and a small number of direct comparisons. On closer examination of earlier meta-analyses, however, the relatively small number of seven trials, that were included in our analysis of large-scaled trials, seems to be the rule rather than the exception. Thus, in the influential systematic review by Bisson & Andrew (Reference Bisson and Andrew2007), for instance, out of more than 80 effect size estimates only three were based on more than seven trials. The meta-analysis by Watts et al. (Reference Watts, Schnurr, Mayo, Young-Xu, Weeks and Friedman2013) shows a similar picture, with a large number of meta-analyses that are based on only few included trials. By combining all available evidence in one analysis we found that the so-called non-specific psychological interventions may be beneficial PTSD treatments; a finding that partly contradicts previous conclusions (e.g. Bisson & Andrew, Reference Bisson and Andrew2007) but conforms to a very recent meta-analysis on present-centered therapy (Frost et al. Reference Frost, Laska and Wampold2014). In the latter paper, present-centered therapy, which has been designed as a control for non-specific intervention effects by Schnurr et al. (Reference Schnurr, Friedman, Foy, Shea, Hsieh, Lavori, Glynn, Wattenberg and Bernardy2003), has been proven to be as effective as trauma-focused and evidence-based interventions.
We therefore do not share the evaluation of de Jonghe et al. of our meta-analysis as being ‘marginally’ relevant ‘at best’. If our findings were confirmed by high-standard trials, they may indeed be of great value for decision making in clinical practice. In this sense, as discussed in our initial report, the findings from our study, which need to be confirmed in adequately designed experimental trials, should be regarded as hypothesis generating. This, however, is true for any meta-analytic finding, as meta-analysis is observational by nature.
Acknowledgments
We thank Prof. Dr. J. Gaab for comments on a previous version of this letter.
Declaration of Interest
None.
