Calling time on risk assessments

INTRODUCTION

Risk assessment continues to loom large in the work of mental health professionals in the UK. In this editorial, I discuss risk assessment as a form of prognostic test. There is an accepted technology for measuring the quality of prognostic tests in medicine and risk instruments (suicide or violence) are generally of much lower predictive validity than would be considered acceptable in other areas of medicine. Patients subject to these tests are both frequently subject to testing without their consent and not guaranteed an outcome better than if they had not been tested, which makes those tests ethically questionable. Government and trust policies frequently go beyond the very limited evidence for efficacy of risk assessments. There are alternative, evidence supported alternatives to risk assessment which would be both safer and more patient centred. This editorial focuses on suicide risk assessment, though many of the same methodological and ethical issues are found in consideration of violence risk assessment: the interested reader is referred to Langan (2010).

STRATEGIES FOR PREDICTION

There are various strategies for generating a predictive test of whether a patient will commit suicide or carry out a violent act. Loosely, these fall into three groups: clinical approaches, making an expert judgement based on largely dynamic factors; actuarial approaches using multiple regression or Baysian methods to sum largely static risk factors into a single predictive value; or fast and frugal heuristics. Fast and frugal heuristics (Gigerenzer & Selten, 2002) have few (official) adherents in psychiatry1: the approach originates with and essentially suggests that in complex situations, reducing the amount of information needed to come to a decision (sometimes radically to a very few items) can improve the outcome of the decision. Actuarial methods (though they have their critics) generally outperform clinical judgements and the arguments for the methodology are strong (Grove & Meehl, 1996).

EVALUATING TESTS

There are a variety of standard approaches for evaluating predictive tests which are independent of the process used to generate the test. These are usually introduced by way of a two by two table (Table 1).

Table 1. Evaluating predictive tests

Positive Predictive Value = true positive fraction TPF=TP/(TP+FP)

Negative Predictive Value = true negative fraction TNF=TF/(TF+FN)

A good test correctly identifies a high proportion of people who go on to commit suicide (enabling an intervention to reduce risk) and a high proportion of people who do not (ensuring that members of the population are not treated unnecessarily with toxic or restrictive treatments).

The first problem is defining the length of time that one is assessing the patient for: a week, a year, ten years? In general, the longer a psychiatric population is followed up, the greater the proportion who have died by suicide: risk increases in proportion to length of follow up.

The second problem is that in real populations, sensitivity and specificity are dependent variables. In practice, this means that increasing the threshold of a test (imagine a suicide questionnaire scored out of ten, where the threshold for ‘high’ is moved from eight to nine) increases the true positive rate, but at the cost of the true negative rate2.

… even if a risk assessment had a sensitivity and specificity of 80% (which probably exceeds those currently available), for every 20 000 patients discharged, 40 would commit suicide, 32 of whom would be identified as high risk. However, in total 4024 patients would be considered to be high risk, 3992 of whom would be false positives. (Geddes, 1999).

The difficulty of predicting suicide is a consequence of the low base rate of suicide (9.21/100 000/year according to the 2010 National Confidential Inquiry) and the low risk multipliers of key risk factors. For example, Harriss et al. (2005) gave the odds ratios of some common risk factors:

• Use of alcohol 1.14

• Widowed/divorced/separated 1.51

• Age 55+ 1.79

Risk of suicide is very unevenly distributed: at the low risk end of the distribution there is a ‘long tail’ with 74% of suicides not in contact with mental health services in the twelve months preceding death (National Confidential Enquiry, 2010). It is difficult for mental health providers to influence outcome in this population. At the other end of the distribution, the very highest risk patients form a tiny minority of all suicides. Powell et al. (2000), in a case control study of 112 suicides, found only two of the patients who committed suicide had a predicted risk of above five percent.

THE ETHICS OF TESTS

Statistical measures of prognostic tests cannot tell us whether or not a particular test is worth adopting or not. They help bring into sharper focus the ethical questions around testing. How many people would a test wrongly identify as suicidal? What would the consequences of being wrongly identified as suicidal for this group? How many suicidal people would the test miss? What would be the consequences for this group?

Sackett & Haynes (2002) guide us through this set of ethical questions with a simple principle: for a test to be of practical use, patients subject to it should fare better than those who are not. This involves thinking about how the results of the test are acted upon. Are patients who are subject to suicide risk assessment likely to have a lower rate of suicide as a result of actions taken as a result of the suicide risk assessment?

In order to answer this question it is helpful to briefly review interventions known to reduce suicide. These interventions fall into two broad groups: those targeted at individuals and those targeted at groups or populations.

In individuals, there are two interventions that have reasonably robust evidence from well conducted meta-analyses that they reduce suicide rates: lithium for bipolar disorder (Cipriani et al. 2005; Baldessarini et al. 2006) and SSRI antidepressants for adults (Barbui et al. 2009; T. P. Laughren, memorandum, 2006: www.fda.gov/ohrms/dockets/ac/06/briefing/2006–4272b1–01-fda.pdf ). In addition, there is good evidence from a well conducted trial that clozapine protects against suicide in psychosis (Meltzer et al. 2003). There is weaker but important evidence for the effectiveness of CBT/DBT in the treatment of suicidality in the form of a meta-anlysis (Tarrier et al. 2008). Lastly, there is evidence that partial hospitalisation for borderline personality disorder (a single trial: Bateman & Fonagy, 1999) may reduce suicide. These last two both suggest a reduction in suicidal behaviour with psychological treatments but do not show a reduction in completed suicide.

One can also target groups or populations in an attempt to reduce suicide. Restricting access to the means of suicide may delay or prevent suicide in vulnerable people. Suicide reductions have been shown with the switch to carbon monoxide free gas in the UK, restricting pack sizes of paracetamol and fitting cars with catalytic converters, restrictions on the use of barbiturates, firearms restrictions and restrictions on pesticides (Mann et al. 2005). Evidence is contradictory as to whether such interventions lead to a reduction in total suicides or not. For example, Sinyor & Levitt (2010) examined the effect of a barrier at a well known suicide spot in Toronto and found that although the barrier reduced the annual rate of suicide from 9.3/year at that bridge to zero, rates of suicide by jumping in Toronto remained unchanged: that is interventions that reduce access to means of suicide may work by displacing (temporally or geographically) the point at which a person commits suicide. A corollary of this effect may be that length of follow up is correlated with measured suicide rate, which is well established (e.g. Hawton et al. 2003). Although there is little direct evidence for the hypothesis that postponing a suicide attempt to allow a therapeutic intervention to be made is effective, it has face validity and is usually recommended by experts as a strategy.

Both of these two groups of interventions are made independently of a risk assessment. In the first group, a treatment is offered on the basis of a diagnosis made. In the second group, an intervention is offered to a population without risk assessment on a patient by patient basis. In other words, no currently available evidenced based intervention to reduce suicide depends on the performance of a suicide risk assessment. Returning to Sackett & Haynes (2002) principle, it would seem that patients do not benefit from having their suicide risk assessed because there is no action that can be taken on the basis of the test that could not be taken without it.

To completely measure suicide risk assessments against the Sackett & Haynes principle, one must also consider whether risk assessments do harm. Risk assessments are often used to justify restrictions on patient autonomy, a practice that has been strongly criticised, for example by Bentall (2010) who expressed four principle objections:

1. 1. Clinicians do not reliably know what is in their patient’s best interest. How can we coerce when we do not know in which direction we should be leading our patients? In the absence of knowing what is in our patient’s best interest, an easier goal (what is in the clinician’s best interest) is often substituted. I have previously argued (Undrill, 2007) that one of the unintended consequences of focussing on risk management of individual patients is the moral hazard that primary risk assessment (the assessment of the patient’s risk of carrying through an act of suicide, self harm or violence) often comes to take second place to secondary risk assessment and the contagion of anxiety that comes in its wake. Secondary risk assessment is essentially reputation management, whether at the level of the clinician (‘I don’t want to go to a coroner’s court, so I’m not going to do anything that could be portrayed as risky’), the provider (‘We need to adopt robust risk management protocols or we will be in trouble with monitoring agencies’) or the NHS higher management (‘In dealing with the media, not being seen to do something about risk is politically unacceptable’).

   Secondary risk management is problematic for many reasons. It draws clinician and organisational effort away from our core task of helping mentally ill people. Every pound spent on secondary risk activity is a pound not spent on an evidenced based treatment; every hour spent on extensive risk assessment procedures is an hour not spent with a patient doing useful psychotherapeutic work. In an era of financial austerity, it is more important than ever to identify unproductive activities and stop doing them. Risk management is intrinsically corrosive of the human relationships that form the foundation of mental health care, because of the element of bad faith that is inevitably introduced by justifying secondary risk activities by falsely portraying them as of benefit to the patient. Secondary risk management at all organisational levels tends to produce action and restrictions of liberty at a cost to the patient.

2. 2. Compelling patients to have treatment is wrong when the treatment is ineffective; to which it might be added that detaining patients in hospital and denying them evidence based treatment for their condition for which they have been detained is just as ethically questionable.

3. 3. Coercion is intrinsically damaging to mental health. Contemporary psychotherapeutic research (e.g. Miller & Rollnick, 2002) repeatedly emphasises self efficacy and self determination as an important part of treatment. Patient satisfaction is inversely correlated with perceived coercion (e.g. Katsakou et al. 2010).

4. 4. Coercion is damaging to therapeutic relationships. Bentall (2010) noted how this in turn affects variables such as the patient’s willingness to engage with services and length of stay in hospital.

Patients are rarely made aware of the issues at stake when they are asked to cooperate with the prognostic test risk assessment. It would be interesting to know how many would consent if the risks and benefits were fully explained to them in the way that is normal in other areas of medicine when a test with serious consequences is proposed.

FROM RISK TO SAFETY

Risk assessment as it is currently practiced in most UK hospitals is of little benefit to patients and may even do harm, yet suicide remains an important health problem. How should this be addressed? By building safe systems in which humane, patient centred and evidenced based care can be reliably and consistently delivered.

The conceptual foundation for safe systems is decoupling risk management decisions from individual patient risk assessments. Risk is then managed by building safety into globally applicable policies, procedures and care pathways wherever possible, both within and outside clinical settings. The change of emphasis from risk to safety is essentially a change of perspective from looking at individual patients to looking at public health and has already begun: patient safety expert Atwul Gawande has called for psychiatry to embrace a public health approach (Moran, 2009) and many of the recommendations of the National Confidential Inquiry into suicides and homicides (2010) are grounded in a universalising, ‘safe structures’ approach. For example, efforts to remove ligature points from all clinical areas are more likely to reduce inpatient suicide than using unreliable methods to assess risk and reduce access to ligature points on a case by case basis. There are other aspects of care from the physical environment to care pathways, handovers and transfers between teams all of which could be designed with global patient safety in mind so that risk assessment on a case by case basis becomes a far less important part of the daily work of clinicians. A systems based approach is not new or innovative and is well established in the literature on safety in other areas of human endeavour (e.g. Reason, 1990).

Safe systems free up staff to focus on core tasks of providing high quality evidence based care to patients without constantly having to address their own reputational risks in systems awash with organisational anxiety over risk. A focus on treating suicidality and its underlying causes is crucial as much of the evidence about effective strategies for reducing suicide focuses on provision of treatment. Currently, evidence based psychological treatments are inconsistently available to people in risk groups. For example, in psychiatric in-patients, one of the highest risk groups, the bulk of care is delivered by nursing staff with little or no training in psychological approaches, often with minimal clinical (as opposed to management) supervision. Many patients who self harm or take overdoses are discharged from general hospitals without any psychiatric follow up, or do not attend the follow up they are offered (Lizardi & Stanley, 2010). Where evidence is lacking, as it is in much of psychiatry as it is in much of medicine, we should be modest and cautious in providing treatments that restrict patients’ liberties or may be actively harmful.

The evidence and the ethics point in the same direction. It’s time to call time on risk assessments and to get back to our core business of treating mentally ill people.