Evaluating risks

Mistakes are inevitable in any branch of medicine, but psychiatry is a particularly risky business (Holloway, 1997). When psychiatrists get it wrong there may serious consequences for their patients, the clinical team and the wider public. The Government introduced a series of initiatives in the 1990s: the Care Programme Approach (1990), the supervision register (Department of Health & Home Office, 1994) and supervised discharge (Secretary of State for Health, 1997). One of the main purposes of this legislation was to minimise the risk psychiatric patients pose to the community. Future service provision will be shaped by clinical governance and the National Service Framework for Mental Health (Secretary of State for Health, 1997), and evaluation and management of risk will become increasingly important.

Evaluating risk is part of everyday practice. For example, we often need to weigh the risk of harmful behaviour against the benefits of discharging individuals from hospital, or the risk of side-effects against the potential therapeutic benefits of a course of treatment. Holloway (1997) highlights one day's risk decisions in the life of a general psychiatrist. Psychiatric practice could be likened to a minefield. The question is, how do we negotiate it? How do we turn Òan impossible burden” into Òa demanding but intellectually rewarding task”(Holloway, 1998)? In this paper I will discuss the concept of risk and different approaches to evaluating it, as well as considering some of the practical aspects of assessing and managing risk.

What is risk?

The word ‘risk’ came into the English language from Spanish or Portuguese, where it referred to sailing into uncharted waters (Williams, 1999). Latterly, it has been used by insurance and finance companies to express how likely a policy holder is to make a claim. Risk is simply the probability or likelihood of a particular event occurring. Perhaps because of human nature, it usually refers to an unwelcome outcome (e.g. disease onset, relapse, death or disability), but it is just as valid a concept for welcome outcomes (prevention, remission or recovery). When discussing risk, the outcome of interest needs to be clearly specified. On an individual level this might include the risk of suicide, side-effects from medication, offending behaviour or violence and aggression. Alternatively, we might wish to consider wider outcomes, such as the potential effects on the public or even the risks to the reputation of individual clinicians and the standing of the clinical service (Snowden, 1997).

Risk tends to be expressed in binary terms, with a person said to be at either ‘high risk’ or ‘low risk’ of a particular disorder. This is because most medical decisions and treatments are binary in nature. Our prescribing options for lithium in the prophylaxis of bipolar disorder (to prescribe or not prescribe) depend on our assessment of the patient's risk of relapse (high or low). In fact, classifications of high- and low-risk groups may be largely arbitrary, since most risks are distributed continuously throughout populations (Rose, 1992). However, it has been argued that such dimensional approaches to risk are not clinically useful (Kraemer et al, 1997).

Risk as a dynamic concept

Risk is not static. It varies between populations and across age ranges. For example, marriage is a risk factor for suicide among teenage girls, but a protective factor among adult women (Bancroft et al, 1975; Hawton, 1986). Even for an individual, identified risks can increase or decrease over time, and the nature of the risks may change (Snowden, 1997). This has important implications for risk assessment, which may be no more than a ‘snapshot’ at one particular moment in time. Risk also varies depending on the stage of the disorder we are considering. The risk factors for a first onset of major depressive disorder are different from those for relapse, which in turn differ from those for remission or recovery.

Expressing risk

The degree of risk can be expressed in several ways. A clinical classification might simply use categories (high and low risk). Measures of risk that can be calculated from epidemiological studies include attributable risk (the disease rate in persons exposed to the risk factor of interest minus the disease rate in unexposed persons) and relative risk (the ratio of disease rate in exposed persons to those not exposed). Attributable risk is the measure of association that is most relevant for making decisions about individuals, but relative risks are more generally available because they can be estimated from a wider range of study designs. Relative risks can be misleading. For example, the relative risk of suicide in the year following self-harm is 100 (Hawton & Fagg 1988). This seems impressive, but because suicide is a rare outcome it means that if self-harm is used as a basis on which to predict future suicide, the prediction will be correct less than 1% of the time. The population attributable risk is the rate of disease (or outcome of interest) in the population minus the rate that would apply if all of the population were unexposed. It measures the potential impact of control measures in a population. For example, it has been calculated that if the USA were able to prevent schizophrenia completely, the amount of crime in the population would decrease by only 3% (Wesseley, 1997).

Quantifying risk

Research suggests that we are not terribly proficient at quantifying and predicting risk. For example, when predicting violent behaviour one study shows that mental health professionals are wrong 95% of the time (Ennis, 1972). It could be argued that we might be better off simply assuming no one will have the (generally rare) outcome of interest, in which case we will be right more often than not (Grubin, 1997). It may also be that specialist skills are not a prerequisite for accurate risk prediction. Teachers have been found to do almost as well as psychiatrists in simulated assessments (Quinsey & Ambtman, 1979). Central to the task of improving the risk assessment skills of clinicians has been the debate regarding the relative merits of actuarial and clinical methods of risk evaluation.

Actuarial approaches to risk

Statisticians use ‘actuarial’ to refer to mathematical techniques they apply to set insurance premiums and pension schemes. In the psychiatric risk assessment literature the term has been used to describe any mathematical means of combining information (Buchanan, 1999).

A recent study of suicide in hospital in-patients provides a good example of an actuarial approach to risk (Powell et al, 2000). When 112 in-patient suicides were compared with 112 controls, five factors were found to distinguish particularly strongly between the two groups: recent bereavement, presence of delusions, suicidal ideation, chronic mental illness and a family history of suicide. Ideally, these risk factors would then have been used to predict suicide in an independent sample, but the authors tested the predictive power of the factors in the sample from which they were derived. This is less than ideal, but the results are still of interest (Fig. 1). Using a cut-off of two or more risk factors to denote ‘high risk’ of suicide, the specificity of the test (an indication of how well it identifies non-suicides) is very good, at 89/90 or 99%. However, its sensitivity (an indication of how well it identifies suicides) is poor, with only 26/97 or 27% being correctly identified. In other words, we will fail to identify almost 75% of suicides using this test. Of course, we can lower our cut-off so that high-risk individuals are defined as having just one or more risk factors, but this will give an unacceptable number of false positives (non-suicidal patients identified as high risk).

Fig. 1 (a) sensitivity (a/(a+c)), specificity (b/(b+d)) and predictive value (a/(a+b)) of a diagnostic test to assess the risk of suicide. (b) Powell et al's (2000) results for psychiatric in-patients

This illustrates a very important point regarding actuarial methods in psychiatry: their usefulness is limited because most of our assessment tools have limited specificity and sensitivity, and the outcomes we are interested in have a low incidence. This means that the positive predictive value of actuarial assessments (the proportion of individuals identified as high risk who are actually high risk) is often disappointingly low, and many patients would need to be treated to prevent one adverse outcome. For example, figures presented by Geddes (1999) indicate that even with a ‘perfect’ treatment (i.e. one that is 100% effective) and a very good risk assessment tool (one with 80% sensitivity and specificity) we would have to treat over 100 high-risk patients following their discharge from hospital to prevent a single suicide.

There are other problems with actuarial techniques (Grubin, 1997). These include a lack of flexibility, poor generalisability across patient groups and limited utility in everyday practice because clinicians often have only fragmentary information on which to base clinical decisions. Actuarial approaches also fail to take into account the individual circumstances of the patient and to provide an explanation of the behaviour.

Newer tools for actuarial assessment have overcome some of these difficulties. One such tool is the iterative classification tree (ICT) for assessing violence risk (Monahan et al, 2000). In a USA study, 106 risk factors in four domains (personal, historical, situational, clinical) were measured at baseline in 939 in-patients. The patients were followed up for 20 months after discharge to investigate which factors were predictive of future violence. The abridged version of the ICT is designed for use in routine practice, and Box 1 shows some of the factors that it considers when predicting risk. The ICT is based on an interactive and contingent model of violence that permits consideration of combinations of risk factors in order to classify a person as high or low risk. The questions are asked according to a tree structure (determined in the original study by statistical consideration of the clustering of risk factors and patients). All subjects are asked a first question. The answer to this determines the next question on the tree and so on until each person is classified into a risk category. One of the advantages over traditional approaches is that this model explicitly acknowledges the fact that violence is an outcome that can be reached by multiple routes. The ICT seems to be superior to existing actuarial risk assessment tools, with only 15% of patients being assigned to an incorrect risk category, but it is still unclear whether the findings are generalisable to the practice of psychiatry in other health care settings.

Box 1 Factors to consider when assessing violence risk using the iterative classification tree (Monahan et al, 2000)

Previous convictions and seriousness of offence

Recent violent fantasies

Drug use by father

Motor impulsiveness

Substance misuse

Legal status

Major psychiatric disorder

Diagnosis of schizophrenia

Anger reaction

Employment status

Recent violence

Loss of consciousness

History of parental fighting

Clinical approaches to risk

Some view the clinical method of risk assessment as merely an informal and unsystematic version of actuarial approaches and describe clinical experience as Òa prestigious synonym for anecdotal evidence” (Grove & Meehl, 1996). It has been argued that clinicians either allocate patients to reference classes or assign weights to variables and combine these weights mathematically, just as is done using actuarial tables, but that the clinicans do it much less well.

In much of the research literature clinical risk assessment is defined by a process of exclusion and refers to any means of prediction that is not mathematical in nature. It is much more than this. It is a person-specific assessment, which takes into account past behaviour and the context in which it occurred (Vinestock, 1996). It refers to a Òbalanced summary of prediction derived from knowledge of the individual, the present circumstances and what is known of the disorder from which he is suffering” (Department of Health & Home Office, 1994). Central to this approach is a detailed understanding of the person's underlying mental state and psychopathology. The risks in each individual case need to be identified and then assessed in terms of frequency and severity. The assessment process should be multi-disciplinary and information should be obtained from all available sources (Snowden, 1997).

Clinical approaches undoubtedly have drawbacks. When clinicians are asked to predict adverse outcomes they almost invariably overpredict. This is probably because of the relative consequences of the two types of mistake that could be made (Buchanan, 1999). For example, in the current risk-averse climate, most psychiatrists would probably rather detain someone who turns out not to be violent (a false positive) than discharge someone who subsequently commits a violent act (a false negative). This inevitably raises a number of ethical issues regarding the erosion of patients' civil liberties. How many false positives is society prepared to accept in the interests of ‘safety’?

Reports from recent inquiries give us a valuable insight into situations where clinical risk assessment has failed to prevent disasters in mental health services (i.e. false negatives). Some of the reasons for failure are listed in Box 2 (Lipsedge, 1995).

Box 2 Why things go wrong with risk assessment (Lipsedge, 1995)

Failure to lend sufficient weight to reports by carers and members of the public about disturbed behaviour

Undue emphasis on the civil liberties of patients

Failure properly to implement the Mental Health Act

Tendency to take a cross-sectional rather than a long-term view of risk

Failure to share information in the best interests of the patient

Practical risk assessment

Perhaps the most sensible approach to risk assessment would be to combine both actuarial and clinical methods, using our knowledge of risk factors from epidemiological studies as well as the patient's individual characteristics, in order to inform our clinical decisions. Perhaps surprisingly this approach has not received much support in the literature, and the consensus seems to be that we need to come down on one side of the fence or the other. This may be short-sighted. Even if actuarial approaches become more sophisticated and categories of risk factor better defined, clinical skills will continue to play an important role in determining whether a patient falls into a particular category or not. As the phenomenon being measured becomes more unusual, the level of clinical skill required to measure it increases (Buchanan, 1999).

A risk assessment framework

It may be that the role of risk assessment is not about making an accurate prediction, but about making informed, defensible decisions (Grounds, 1995). The content of risk assessment varies according to the adverse outcome being assessed (e.g. the risk of harm to self or others) and has been comprehensively covered elsewhere (Alberg, et al, 1996; Royal College of Psychiatrists, 1996; Vinestock, 1996).

Moore (1995) suggests a framework that can be applied to most risk assessments in psychiatry (Box 3). The behaviour to be predicted must be defined and each behaviour should be assessed individually, as each is likely to involve different risk factors. The assessing team needs to distinguish clearly between the probability (likelihood) of the behaviour and its cost (potentially serious consequences). Assessors must be aware of the possible sources of error in the assessment, arising from the patient, the assessors themselves and the context of the assessment. The interaction of internal factors (e.g. attitudes, drives, needs, controls) and external environmental factors (e.g. demands, constraints, stressors) in producing the target behaviour should be considered. A check should be made of whether all necessary information has been gathered. If it has not, what additional information is needed and where might it be obtained? A longitudinal perspective should be adopted, with some prediction of the factors and circumstances likely to increase or decrease future risk. Key interventions should be planned and a decision made about whether to involve other professional groups.

Box 3 A risk assessment framework (Moore, 1995)

Define the behaviour to be predicted

Distinguish between the probability and the cost of the behaviour

Be aware of the possible sources of error in the assessment

Take into account the influence of both internal and external factors on the behaviour

Check that all the necessary information has been gathered

Predict factors likely to increase or decrease future risk

Identify when other professionals or agencies need to be involved

Plan key interventions

Too many decisions, too little time

Sometimes the realities of clinical practice mean that we do not have as much time as we would wish to make clinical decisions. We often have to act in less than ideal circumstances with less than comprehensive information. Three questions might help us to prioritise risk decisions (Taylor 1995): What is the seriousness of the risk? What is the imminence of the risk? What is the probability of the risk becoming actual?

Managing risk

In its simplest sense, clinical risk management describes the development of strategies to reduce the severity and frequency of identified risks and adverse outcomes (Vincent, 1995). In psychiatry this might involve a clear record of the role and responsibilities of each professional involved with a patient, clinical audit of adverse incidents and predetermined plans of action when concerns arise (Snowden, 1997). Good-quality medical records, thorough note-keeping and open communication are key aspects of effective risk management, as is compliance with statutory requirements. Clinical guidelines, checklists and management protocols have the potential to reduce the risk associated with practice. On an organisational level, risk management requires a service that is adequately staffed with personnel who are fully trained in the current best clinical practice. Risks also need to be shared, with junior staff having adequate access to consultant advice.

Risk substitution

It could be argued that minimising a risk by effective risk management simply leads to a compensatory increase in other risk factors, and so is largely a waste of time. For example, it may be that changes to gun laws in the USA in the 1980s led to a compensatory increase in other methods of suicide. An alternative to this ‘risk substitution hypothesis’ is the ‘risk reduction hypothesis’. It has been estimated that 6700 suicides were prevented by the detoxification of domestic gas in the UK in the 1960s. These concepts are impossible to disentangle (Gunnell & Frankel, 1994) and there is little firm evidence one way or the other. However, policy-makers continue to be persuaded by the risk reduction hypothesis, as evidenced by recent legislation to limit the over-the-counter availability of paracetamol in the UK.

Evaluating risks and making decisions

Much of clinical practice involves us in evaluating and balancing the risks of different courses of action. ‘Decision analysis’ was originally developed in the business world and is an explicit quantitative approach to examining difficult decisions. Its aim is to enable clinicians to make the best decision for individuals or groups of patients (Hatcher, 1995).

Decision analysis involves assigning a probability to each potential clinical outcome. These probabilities represent the doctor's ‘best guess’ about how likely a particular outcome is, and are based on clinical features and data from relevant clinical studies. A utility is also assigned to each outcome. This represents a patient's preference for one clinical outcome over another. Utilities are given values from 0 to 1 (0=worse possible outcome, e.g. death; 1=best possible outcome, e.g. perfect health). The various decision points and their consequences, along with their associated probabilities and utilities, are mapped to form a decision tree. This is a visual representation of the decisions available. The probabilities and utilities of the various decisions are combined to determine the best treatment. The best treatment is the one with the highest expected utility (i.e. the treatment most likely to provide the optimal outcome for the patient). Figure 2 shows a simplified decision tree for prescribing antipsychotics in first episode schizophrenia with respect to the risk of tardive dyskinesia and recurrence of the disorder. In this example, the expected utility (value) of using antipsychotics or not is calculated by summing the product of probabilities and utility for all outcomes distal to the first branch of the tree. So the expected utility of not using antipsychotics is (0.7 × 0.45) + (0.3 × 1.0)=0.615; that of using antipsychotics is (0.15 × 0.58 × 0) + (0.15 × 0.42 × 0.75) + (0.85 × 0.58 × 0.45) + (0.85 × 0.42 × 1)=0.626. In this case it is better to treat with antipsychotics than not to.

Fig. 2 Decision tree for prescribing antipsychotics in first-episode schizophrenia with respect to risk of tardive dyskinesia and recurrence of disorder (adapted from Hatcher, 1995). Probabilities (P) of different outcomes are based on research evidence and the clinician's ‘best guess’. Utilities were determined in this case by asking three clinicians. Utility=0 indicates the worse possible outcome, and utility=1 the best possible outcome

The use of decision analysis to evaluate risks in psychiatry has its problems. The quality of the psychiatric research database is poor – the evidence to guide us in certain situations simply does not exist. There may be difficulties in generalising decisions to other patients. The value, or utility, ascribed to outcomes is arbitrary and, especially when considering risk, may vary between those involved. The utility to the patient of a particular outcome may differ from its utility to the clinical team or wider society. Nevertheless, decision analysis might prove a useful adjunct to current methods of risk evaluation and certainly has a role to play in teaching, audit and research (Hatcher, 1995).

Alternatives to a ‘high-risk’ approach

Psychiatry is currently preoccupied with the concept of risk, and practice is based on what is essentially a ‘high-risk’ strategy. This involves applying risk assessment to a population, and then focusing intervention on those identified as being at greatest risk of poor outcomes (Rose, 1992). The advantages of this strategy are clear. It offers a cost-effective use of resources, since only a minority of individuals is targeted. It avoids unnecessary and potentially harmful treatment of those who do not need it. It leads to interventions that are more likely to be tailored to the needs of the individual. Unfortunately, there are serious problems with an exclusively high-risk strategy. First, those in the lower-risk, higher-volume group are excluded even though they represent a larger absolute number of those with poor outcomes and, therefore, a greater population burden. Any intervention targeted at the high-risk group alone may make only a small contribution to reducing the overall incidence of the disease in the population. Second, instruments screening for the high risk may be expensive, impractical or lack predictive power. Third, the distinction between high- and low-risk groups may well be an artificial one.

The current management of deliberate self-harm is an example of a high-risk strategy, and it illustrates the disadvantages of this approach (Kapur & House, 1998). First, targeting high-risk groups results in large numbers of supposedly low-risk individuals being ‘defined out’ of care, reducing the impact of suicide prevention strategies. For example, those identified as being at high risk using one risk assessment scale account for only 26% of cases of future suicidal behaviour, the much larger low-risk group accounting for the remainder. With intervention restricted to the high-risk group, even assuming that it is totally effective (which is improbable), we will reduce the overall rate of suicidal behaviour by at most one-quarter. Second, available risk measures are poor at predicting repeated suicidal behaviour (positive predictive value 25% at best). Third, suicide risk may be continuously distributed in a population, in which case a dichotomous distinction between high- and low-risk groups is not valid.

The population-based strategy of prevention is an alternative to the high-risk approach, and it involves targeting whole populations rather than just vulnerable individuals. This is potentially powerful because of the number of individuals involved; even very small population shifts can have large effects. To use an example from general medicine, a fall of just 3% in the mean population blood pressure reduces the population prevalence of hypertension by 25% (Rose, 1992). Population approaches are also radical since they seek to address underlying causes of a phenomenon rather than just its external manifestations. Again using the example of deliberate self-harm, a population strategy might involve: limiting the availability of methods of self-harm; a review of economic and social policy; and changing our clinical management to offer interventions to everyone following an episode of deliberate self-harm and not just those at high risk.

However, the implementation of population approaches may be unacceptable or prohibitively expensive, and is not always compatible with our medical model of managing health problems. A combination of population and high-risk strategies might be a more feasible and effective option (Kapur & House, 1998). For deliberate self-harm, this could take the form of providing a basic level of intervention for all patients (e.g. an emergency card listing contact numbers) and then using clinical and actuarial methods of risk assessment to identify patients who may benefit from more intensive treatment (e.g. interpersonal or problem-solving therapy).

Conclusion

In psychiatry, as in other fields of medicine, there has been a temptation to view risk as an entirely negative concept, but risk has its positive aspects as well (Holloway, 1997). Risk can be viewed as a mobilising force for change in society (Williams, 1999), and no risk may mean no progress. This is all very well, but it does not help us much on a practical level. What skills do we need to evaluate and manage risks and how might we best impart these to trainees? The traditional apprenticeship model is quite good at teaching the necessary clinical skills, provided that the senior clinician explains how decisions are made and encourages a questioning attitude. However, a clinical skills apprenticeship is not enough. It needs supplementing. Psychiatrists should have knowledge of epidemiological, actuarial and population perspectives of risk in order to deal more effectively with the risks that are an inherent part of clinical practice.

Multiple choice questions

1. 1. Risk:

   1. a refers to the probability of a particular event occurring

   2. b is a static concept

   3. c is always negative

   4. d is usually expressed in categorical terms

   5. e is relevant to psychiatric practice.

2. 2. Actuarial approaches to risk assessment:

   1. a consider the combined effects of groups of risk factors

   2. b are extremely good at predicting risk

   3. c may lack clinical relevance

   4. d have been shown to be superior to simple clinical judgement

   5. e are continuing to evolve.

3. 3. Clinical approaches to risk assessment:

   1. a are based purely on guesswork

   2. b result in overestimates of risk

   3. c are extremely good at predicting risk

   4. d consider individual factors and past history

   5. e are incompatible with actuarial methods.

4. 4. Risk management:

   1. a is not applicable to psychiatry

   2. b is not influenced by staff or resource levels

   3. c may be enhanced by clinical guidelines

   4. d may involve the investigation of adverse incidents

   5. e may be improved by use of decision analysis

5. 5. Population-based strategies of risk prevention:

   1. a target vulnerable individuals only

   2. b are more powerful than high-risk approaches

   3. c may be expensive to implement

   4. d may not be compatible with the medical model of health care

   5. e involve assigning individuals to high- or low-risk groups

MCQ answers

1		2		3		4		5
a	T	a	T	a	F	a	F	a	F
b	F	b	F	b	T	b	F	b	T
c	F	c	T	c	F	c	T	c	T
d	T	d	T	d	T	d	T	d	T
e	T	e	T	e	F	e	T	e	F