Low-risk and high-risk groups matter in suicide risk

Dear Sir,

We read with interest the article by Large (Large et al., 2018) who reported on suicide risk assessment from a heterogeneous selection of papers, either cohort or case control (Q-value = 143, df (Q) = 17, p value < 0.001, I ² = 88.1%). Based on their analyses, they questioned the utility of risk assessment, as has another recent narrative review (Carter and Spittal, 2018). However, we have found many services mandate risk assessment tools. This led us to search for risk assessment measures that published sensitivity, specificity and prevalence rates for completed suicide, which allowed us to estimate Bayesian post-test predictive values (PPV), and to perform a meta-analysis on the pooled sensitivity and specificity for these suicide risk assessment scales in relation to completed suicide.

We searched Medline, PubMed and Embase for papers that reported psychometric properties of suicide risk assessment. We extracted, directly from the paper, or indirectly from the published sensitivity, specificity and mortality data, the true positive, false negative, false positive and true negative rates. We used these to estimate the pooled sensitivity, specificity using a fixed effects model. We estimated heterogeneity using Cochrane's Q (Cochran, 1950) and I ² (Hedges and Pigott, 2001). A Standardized Receiver Operation Curve (SROC) was created, along with 95% confidence intervals for sensitivity and specificity using a bivariate analysis. We also estimated the PPV, from Bayes’ formula.

Post-hoc we divided the papers into low- and high-risk populations from the clinical descriptions of the populations, and the published suicide rates. The high-risk groups all had suicide death rates over one percent. This analysis was performed using the mada (Doebler, 2015) package in R version 3.4 (R Development Core Team, 2017).

Our search identified 495 papers, of which 109 remained after screening abstracts, but only seven papers met our criteria. Of the four high-prevalence papers, one reported on persons who had made a suicide attempt within 1 month (baseline suicide rate 8%) (Stefansson et al., 2012). A second paper reported on those admitted after a suicide attempt to a suicide research institute where the baseline suicide rate was 4% (Nimeus et al., 2000). A third paper was in people with schizophrenia with active suicidal ideation or command hallucinations to suicide (baseline suicide rate 1.3%) (Ayer et al., 2008). A fourth paper combined a smaller pilot in a high-risk group with two cohorts that presented to UK hospitals (Pallis et al., 1984). The suicide rate in the larger of these samples was 1.5%. The other three low prevalence papers were larger. Two were multicenter studies assessing people in emergency department after self-harm (Steeg et al., 2012; Cooper et al., 2006), with suicide rates of 0.08% and 0.06% and the other was a 20 year series from the Beck Institute (Beck et al., 1999), which reported a suicide rate of 0.02%.

The PPV ranged from 0.3 to 24%. The three datasets with low prevalence had PPV from 0.3 to 1.8%: in the high-prevalence groups the range was 13–24%. Similarly, the Numbers needed to Intervene (NNI) ranged from 4.2 to 325: the low prevalence groups had an NNI between 28 and 325 and the high-prevalence group had an NNI between four and seven.

The pooled data showed remarkably low heterogeneity (Cochrane's Q = 12, I ² = 0). The pooled sensitivity was 0.74 (0.63–0.83), specificity 0.54 (0.35–0.72). The pooled diagnostic odds ratio was 3.9 (1.7–9.4). The sensitivity analysis showed a significant difference using a bivariate likelihood ratio test (χ² = 7, 2 df, p = 0.02). The pooled sensitivity in the high-risk group was 0.89 (0.53–0.98) while in the low-risk group was 0.72 (0.54–0.89). The specificity also differed between high-risk 0.80 (0.48–0.96) and low-risk groups 0.30 (0.08–0.69). The area under the curve was estimated at 0.90 in the high-risk group, and 0.69 in the low-risk group (see Fig. 1).

Fig. 1. SROC for low and high-risk populations, showing in the low-risk group the 95% CIs.

As in all meta-analyses, there are difficulties in obtaining all data: this review was limited to the English language, we were unable to double code all papers, and the number of data points is few. It is of note that our search does not include any papers within Large's analysis, and this may be a result of our requirement that a published scale was used.

The Bayesian analysis indicates that there is a limited clinical utility to using rating scales for predicting suicide death risk. Our conclusion is that in the high-risk group, risk stratification may have some utility, but in general clinical work, or in public health psychiatry, the base rate of suicide is such that risk assessment may indeed be risky (Mulder et al., 2016).

We suggest that this small analysis is further evidence of the lack of utility of risk assessment scales for suicide: the use in high-risk groups is questionable, and it is even more questionable within general psychiatric services. It may be better to focus on developing therapeutic relationships with patients presenting with suicidal thoughts/acts (Gale and Glue, 2018), and designing safety plans and follow-up (Stanley et al., 2018), than to continue to concentrate on estimating the risk of completed suicide.