End of the road for treatment-as-usual studies?

Summary

Using treatment as usual (TAU) in trials has obscured the repeated finding that assertive outreach has never reduced hospitalisation when compared with treatment by multidisciplinary teams (community mental health teams, CMHTs). Its use has delayed recognising that CMHTs are the more cost-effective, evidence-based approach. The term should be abandoned and trials should compare two equally well-defined services.

Killaspy et al's ¹ short report in this month's issue (pp. ) ends with the long-overdue conclusion ‘CMHTs are able to prevent admissions as successfully as ACT teams using fewer contacts… we question the continuing investment in ACT in the UK…’ Why this conclusion took so long to be published deserves careful consideration and has important lessons for the conduct of community psychiatry research. It also probably has important lessons for mental health policy makers, but that lies beyond the scope of this editorial.

There should be nothing surprising about Killaspy et al's conclusion. Evidence that traditional UK community mental health teams (CMHTs) perform as well as assertive community treatment (ACT) teams in the care of the severely mentally ill has been abundant for several years now. Just when Marshall & Lockwood's Cochrane meta-analyses of ACT and case management ^2,3 was providing the evidence base for a wholesale reform of UK mental health services (ushering in the establishment of over 300 ACT teams), ⁴ two major UK trials were casting doubts on its wisdom. The UK700 trial ⁵ demonstrated that there was no reduction in hospitalisation (none at all) in the intensive arm of the study compared with the control arm. The PRiSM study ⁶ was quasi-experimental but benefited from an epidemiological framework and also found no differences between its community models. Both studies attracted vigorous challenges, questioning both their methodologies and the quality of their services. ^7–9 There was a genuine sense of outrage – how could these studies fail to replicate the findings of Stein & Test, ¹⁰ Hoult et al ¹¹ and Rosenheck et al? ¹² That the failure of subsequent studies to match early effects is an almost universal finding across medicine ¹³ seems not to have been any consolation.

Worse was to follow. Not a single large rigorous randomised controlled trial of ACT has been published since to contradict the UK700 and PRiSM trials by demonstrating substantial reductions in hospital care; not even small, quasi-experimental studies do so when the comparator is an established multidisciplinary CMHT. The reasons for these results are now clear. They were established in a careful metaregression analysis of over 30 published trials supplemented by extra data from the trialists (including patient-level data and split-site data) plus a careful estimation of ‘model fidelity’. ¹⁴ Assertive community treatment only reduces hospital care when the baseline level of hospital care is relatively profligate; UK services have been operating for decades at the level that is not reduced by ACT. They have also generally had established CMHTs as the comparator. ¹⁵ The core features of experimental community care teams reported by trialists internationally over several decades ¹⁶ are easily recognisable in standard UK practice.

Hospitalisation as an outcome has been criticised as reductionistic. However, it is a good proxy measure for relapse within a randomised controlled trial (when the thresholds for admission should be the same in both arms although highly variable between trial sites). ¹⁷ It benefits from being consistently recorded and has high face validity. Clinical outcome scores have rarely distinguished successfully between ACT and controls. There has been an increasing tendency to emphasise less objective measures, such as patient satisfaction or adherence, ¹⁸ when differences in hospitalisation are not demonstrated. These are regularly quoted to ‘soften’ the message of no reduction in the primary outcome of hospitalisation. The original REACT study concluded that ‘CMHTs are able to support… serious mental illness as effectively as ACT teams, but ACT teams may be better at engaging clients and may lead to greater satisfaction with services.’ ¹⁹ Why is there a perceived need to soften this message? Why were the UK700 and REACT studies viewed as having a ‘negative result’ requiring explanation?

The curse of TAU

If any of these trials had been conducted by a health economist it is hard to believe they would have been reported as negative. A trial that showed that the same outcome could be achieved at a half to two-thirds of the cost would constitute a major positive finding and one that undoubtedly would have been taken seriously by the National Institute for Health and Clinical Excellence and policy makers alike. Why have they not been reported in this way?

Our appreciation of these ACT trials has been limited by our preconception of ACT as ‘the’ intervention, and the control (treatment as usual, TAU) as simply a necessary structure to facilitate the trial. We pay it almost no attention, not even enough to define it properly. A proposal ²⁰ that the control condition in community psychiatry trials should be at least as well characterised as the experimental condition in publications has had no demonstrable effect. I still regularly review numerous small studies comparing ACT against ‘TAU’ where it is quite impossible to gain even the most remote understanding of what is being provided as TAU. However, the real problem with the proposal is that it did not go far enough.

The problem is in thinking of TAU as a control at all, not just its inadequate characterisation. Far from being an inert control like a placebo, TAU is a very active comparator and, as the metaregression reported above ¹⁴ suggests, a very variable and potent one. When we conduct these community psychiatry studies of complex interventions, we are comparing two interventions and should treat them equally. Clinical equipoise is accepted as an essential precondition for conducting ethical trials and yet our handling and interpretation of the data rarely reflect this. We interpret the data through the prism of the experimental intervention rather than viewing both interventions as equal candidates for endorsement. The UK700 trial described the two arms relatively well but still failed to highlight its finding that standard case management achieved the same results for half the input. In effect, standard case management was twice as efficient and should have been strongly preferred. Our paper on the UK700 trial concluded rather weakly that our results ‘lend little support to the view that simply increasing the number of staff will produce major benefits.’ It called for some ‘greater attention to content of care… rather than its form’. The conclusion, painfully obvious in retrospect, was that standard case management is just as effective as intensive case management for half the cost in this patient group. There was, therefore, no mandate for wholesale change. This has now been roundly confirmed with the national introduction of 300 ACT teams leading to no reduction in bed usage. ²¹

Killaspy et al are to be commended for their conclusion, which the UK700 study could, and probably should, have arrived at 10 years ago. The broader lesson is the need for journals and researchers to adopt greater discipline. This involves ensuring that future trials of complex interventions are conceived of, interpreted as, and then reported as a comparison of two interventions. This is not an easy thing to achieve. Most of us can claim to be in clinical equipoise when we initiate a study. We are genuinely uncertain of the superiority of one approach over the other because of the limitations of the evidence base. However, few of us are utterly neutral on the question. It is hard to imagine committing several years to a clinical trial of an intervention with no personal investment in it. This is a particular problem well recognised in community mental health service research. ¹⁸

Killaspy et al, like the UK700 research team, embarked on their original study ¹⁹ presumably anticipating a superiority for ACT. Like the UK700 team, they were probably initially disappointed and puzzled by their results. However, had their study not been called ‘REACT: randomised evaluation of assertive community treatment in north London’ but something like ‘A randomised trial comparing ACT and CMHT care in north London’ their results would have been neither puzzling nor disappointing. They would probably have been presented quite differently with no need to emphasise minor differences in user satisfaction and follow-up. Their important and convincing conclusion in that first paper ¹⁹ would have been that CMHTs deliver equally effective care for a fraction of the cost in terms of workforce and disruption.

Community psychiatry research cannot control its context. It is often aiming at a moving target as services evolve and it has to incorporate complex aspects of social sciences. Consequently, it needs to be especially rigorous in its methodologies. We should remove TAU from our scientific vocabulary in this contested area. It carries an unwarranted implication that there is some consistent background practice against which any new intervention can be tested. This is not true. It is particularly misleading when research results are applied across different healthcare systems. Removing it would clarify our thinking and improve the scientific quality of our publications. The more balanced understanding and interpretation of data that would flow from treating the two arms of trials genuinely as equals would vastly improve our understanding of services. It might also generate some more rational policy-making. Both are urgently needed.