Letter to the Editor: Comments on ‘Is there an excess of significant findings in published studies of psychotherapy for depression?’

Flint and colleagues make use of a powerful statistical approach to suggest that there is significant positive bias in the expected distribution of published trials of psychotherapy (Flint et al. 2014). They draw a parallel with the bias we know exists in the drug treatment literature and imply that it is again publication bias that explains the finding. In other words, withholding publication has deflated the number of non-significant results.

Indeed, for drug trials conducted by companies (the majority), we know this is the case because the results of negative trials are submitted to the regulatory authorities and can be identified if they are not published. Much has been made of this in previous reviews (Turner, 2013).

However, publication bias is not necessarily the explanation for the findings in psychotherapy trials. We know that the quality of most psychotherapy trials falls far short of what is normal in drug trials (Huhn et al. 2014). Thus, a variety of biases operate to inflate the number of significant results in psychotherapy trials before publication. These include strong allegiance bias, single blind designs and a flexible approach to statistical treatment of outcomes. The result may well be the distorted distribution of results that Flint and colleagues identify in the published literature. A simple prediction is that significant numbers of unpublished negative studies simply do not exist for psychotherapies.

In conclusion, if bias distorts the generation of data rather than simply its reporting, we are presumably uncertain what a true effect size is for psychotherapy.