11 - Error analysis and model validation

Summary

The aim of science is not to open the door to infinite wisdom, but to set a limit on infinite error.

Bertolt Brecht

Introduction

In this chapter we show how the performance of a single learning machine can be evaluated, and how pairs of machines can be compared. Our focus will be on evaluating the prediction error of a machine, that is, how often it places a subject in the incorrect group, case or control, say. We immediately state that the problem of estimating the accuracy of a machine designed for predicting a continuous outcome (temperature, say) is a different and ultimately harder question. A very brief discussion of error analysis for continuous outcomes is given at the end of the chapter, but as stated in Chapter 2, we don't spend nearly enough time on this important problem.

After covering prediction error for a single machine we then examine how any pair of machines can be evaluated: is one significantly better than the other? This kind of paired analysis applies to the comparison of one familiar statistical engine, say logistic regression, with a nonparametric, nonlinear prediction engine such as Random Forests. In this case the comparison is between a big, somewhat hard-to-understand machine and a little, relatively well-understood one. Our analysis of single machines and sets of machines will introduce three ideas.