This chapter covers the perceptron, the simplest neural network architecture. In general, neural networks are machine learning architectures loosely inspired by the structure of biological brains. The perceptron is the simplest example of such architectures: it contains a single artificial neuron. The perceptron will form the building block for the more complicated architectures discussed later in the book. However, rather than starting directly with the discussion of this algorithm, we will start with something simpler: a children’s book and some fundamental observations about machine learning. From these, we will formalize our first machine learning algorithm, the perceptron.

Starting with the perceptron, in Chapter 6 we discuss the functioning, the training, and the use of neural networks. For the different neural network structures, the corresponding script in Matlab is provided and the limitations of the different neural network architectures are discussed. A detailed discussion and the underlying mathematical concept of the Backpropagation learning algorithm is accompanied with simple examples as well as sophisticated implementations using Matlab. Chapter 6 also includesconsiderations on quality measures of trained neural networks, such as the accuracy, recall, specificity, precision, prevalence, and some of the derived quantities such as the F-score and the receiver operating characteristic plot. We also look at the overfitting problem and how to handle it during the neural network training process.

Inspired by the human brain, neural network (NN) models have emerged as the dominant branch of machine learning, with the multi-layer perceptron (MLP) model being the most popular. Non-linear optimization and the presence of local minima during optimization led to interests in other NN architectures that only require linear least squares optimization, e.g. extreme learning machines (ELM) and radial basis functions (RBF). Such models readily adapt to online learning, where a model can be updated inexpensively as new data arrive continually. Applications of NN to predict conditional distributions (by the conditional density network and the mixture density network) and to perform quantile regression are also covered.

This chapter is not intended to be a complete discussion of machine learning. We concentrate on a small number of ideas, and emphasize how to deal with very large data sets. Especially important is how we exploit parallelism to build models of the data. We consider the classical “perceptron” approach to learning a data classifier, where a hyperplane that separates two classes is sought. Then, we look at more modern techniques involving support-vector machines. Similar to perceptrons, these methods look for hyperplanes that best divide the classes, so that few, if any, members of the training set lie close to the hyperplane. We next consider nearest-neighbor techniques, where data is classified according to the class(es) of their nearest neighbors in some space. We end with a discussion of decision trees, which are branching programs for predicting the class of an example.