Avoiding errors about ‘margins of error’

When discussing actuarial risk assessment instruments (ARAIs), Hart et al (2007) acknowledge that ‘prediction’ may refer to probabilistic statements (e.g. a ‘prediction’ that an individual ‘falls in a category for which the estimated risk of violence was 52%’: p. s60). For unclear reasons, however, the authors seem to value only predictions with right or wrong outcomes. They therefore regard statements about future behaviour of large groups (where one can be almost certain that the fraction of persons who act a certain way will fall within a narrow range of proportions) as potentially ‘credible’, but predictions for individuals as meaningless.

If the purpose of risk assessment is to make choices, then well-grounded probabilistic predictions about single events help us. Suppose we conclude that it is legally and ethically acceptable to impose preventive confinement upon individuals in ARAI categories with estimated recidivism rates above a specified threshold. This policy entails making ‘false-negative’ and ‘false-positive’ decision errors. We recognise, however, that unless we are omniscient perfection is not an option and ARAIs simply help us make better decisions than we otherwise could.

How do ‘margins of error’ in estimated recidivism rates affect our decision process? Hart et al believe their ‘group risk’ and ‘individual risk’ 95% confidence intervals speak to this problem. Their group intervals are standard confidence intervals for estimated population proportions based on random samples. If the threshold lies outside the group risk confidence interval for a category, then we can be reasonably certain that a decision we make concerning someone in that category is the same decision we would make if we knew the true recidivism rate for that category. If the threshold falls within a category's group risk confidence interval, then our estimate quite possibly might lead to the ‘wrong’ decision. Statistical decision theory (Berger, 1985) shows, however, that it is still a sensible strategy to choose whether to confine a member of a category based on which side of the threshold our estimated risk falls.

Hart et al talk about ‘individual risk’ as though it is something different from category (or ‘group’) risk. Yet if all one knows about an individual is his membership of a risk group, what can ‘individual risk’ mean? The authors do not say. If ‘individual risk’ refers to believed-to-exist-but-unspecified differences between individuals within a category, such differences should not affect choices by a rational decision-maker. The 95% CIs for ‘individual risk’ pile nonsense on top of meaninglessness. Hart et al describe the replacement of ‘n’ by ‘1’ in the Wilson (1927) formulae as ‘ad hoc’, but this substitution makes no sense when the basis for the estimated proportion is an n-member sample. With ‘1’ in place of ‘n’, the formulae just don't mean anything.

Using ARAIs raises serious moral problems as well as the valid scientific questions that Hart et al mention. But in faulting the capacity of ARAIs to address an unspecified quantity called ‘individual risk’, and in dressing up this notion with misapplied formulae for confidence intervals, Hart et al ultimately create a muddle.