The mathematics of risk assessment for serious violence

Sir: I am pleased that Dr Harry Kennedy has picked up on the issue of prediction of rare events that I mentioned in my paper on inquiries after homicide (Szmukler, Psychiatric Bulletin, January 2000, 24, 6-10; Kennedy, Psychiatric Bulletin, June 2001, 25, 208-211). He makes an important point concerning an assumption in analyses like mine that clinical interventions do not substantially affect rates of serious violence. I will turn to this in a moment.

But first I want to draw attention to Dr Kennedy's use throughout his calculations of a theoretical predictive test for serious violence having a ‘sensitivity’ of 0.9 and ‘specificity’ of 0.9. In my paper I called such a test “wildly unrealistic”. In the real world, a test with a ‘sensitivity’ of 0.52 and a ‘specificity’ of 0.68 is closer to the mark (Buchanan & Leese, 2001). Using these figures the ‘positive predictive value’ (the proportion of positive predictions that turn out correct) for base rates of violence in the patient population of 1%, 5%, 10% and 20% are 0.02, 0.08, 0.15 and 0.29, respectively. (These can be readily calculated using a probability tree method that I have described elsewhere (Szmukler, 2001.)

This means that if violence occurs in say 5% of a patient population, the predictive test will be wrong 92 times out of 100. In an inner-city community mental health team setting we found around that frequency of patients committed an act of violence against persons in a 6 month period (Shergill & Szmukler, 1998), with the vast majority of these incidents not causing serious injury. On the other hand, there is evidence that serious violence in patients with schizophrenia resulting in conviction in a higher court occurs in about 0.5% of males and 0.05% of females — over a 3 year period (Wallace et al, 1998). Here the ‘positive predictive value’, as for homicides, is quite useless; the prediction will be wrong more than 99 times out of 100.

However, Kennedy is right in pointing to a significant caveat concerning these analyses. There are no controlled trials that allow us to evaluate the extent to which psychiatric interventions, including custodial ones, prevent incidents of serious violence. Thus we cannot know what the ‘true’ population base rate might be if clinicians never intervened to prevent them. But is there any reason to believe it would be much higher? Do changes in mental health services, for example, result in significant changes in the rate of serious violence in people suffering from mental illness? There is little to go on. In Victoria, Australia, despite major changes in service configuration, the relative risk of violent offending by patients with schizophrenia compared to controls did not change between 1975 and 1985 (Mullen et al, 2000). I know of no better evidence on the subject. Are these events rare because services are effective in making them so, or are they just rare (as they are in the non-patient population)? We can't know for sure, but the latter must be far more likely. Even if serious violence in males with schizophrenia, without clinical interventions, was 10 times greater than found by Wallace et al, and occurred in 5% instead of 0.5%, the ‘positive predictive value’ of our real world test would still only be 0.08.

Kennedy refers to ‘stratification’ of risk: pick a very high-risk group and focus on them. The cost of doing this is that you then miss the majority of cases who will later be violent. An excellent example concerns the prediction of in-patient suicide, also a rare but tragic event (Powell et al, 2000). The investigators could define a group of patients with all five identified risk factors in whom the probability of suicide was almost 40%. Unfortunately only one out of the 97 eventual suicides was at this level of risk.

If the risk of serious violence could be eliminated by a simple low-risk intervention, such as giving an aspirin, one might be able to put an argument to support the enforced treatment of say 10 or 20 patients to prevent one act of serious violence. However, the interventions we are talking about often involve compulsory treatment or detention for protracted periods of time. The implications of risk assessment are thus extremely serious. Claims for its validity need much stronger evidence than we have so far seen. To me, the mathematics of rare events indicates we are unlikely to ever see it.