Authors' reply

Actuarial risk assessment instruments (ARAIs), constructed using data from known groups, are used to make life-and-death decisions about individuals. How precisely do they estimate risk in individual cases? The 95% CI for proportions, which evaluates the precision of risk estimates for ARAI groups, cannot be used for individual risk estimates unless one makes a very strong assumption of heterogeneity - that ARAIs carve nature at its joints, separating people with perfect accuracy into non-overlapping categories. No one, not even those who construct ARAIs, makes this assumption. So, we ask again, what is the precision of individual risk estimates made using ARAIs?

Mossman & Sellke criticise us for inadequately defining ‘individual risk’ and for using an ad hoc procedure to estimate the margin of error for individual risk estimates, which they opine served only to ‘pile nonsense on top of meaninglessness’.

We must plead guilty to some of the charges levelled by Mossman & Sellke - indeed, we did so in our paper, acknowledging the conceptual and statistical problems with the approach we used. In our defence, we claimed duress: because developers used inappropriate statistical methods to construct ARAIs, we could not use appropriate methods to evaluate them. Violent recidivism was measured in the ARAI development samples as a dichotomous, time-dependent outcome, and so the developers ought to have used logistic regression or survival analysis to build models; if they had, one could directly calculate logistic regression or survival scores for individuals and their associated 95% CIs.

But we also plead that these charges are irrelevant to our conclusion. As we discussed, to reject our findings that the margins of error for individual risk estimates are large is to acknowledge that they are either unknown or incalculable. Regardless, the current state of affairs is unacceptable for those who seek to use these tests in a professionally responsible manner or argue in favour of their legal admissibility. We urge ARAI developers to recalibrate their statistical models in a way that permits direct calculation of individual risk estimates and their precision or to make their data publicly available so others may do so.