Foreword

Summary

July 1997 saw the start of a six month international research programme entitled Neural Networks and Machine Learning, hosted by the Isaac Newton Institute for Mathematical Sciences in Cambridge. During the programme many of the world's leading researchers in the field visited the Institute for periods ranging from a few weeks up to six months, and numerous younger scientists also benefited from a wide range of conferences and tutorials.

Amongst the many successful workshops which ran during the six month Newton Institute programme, the one week workshop on the theme of Online Learning in Neural Networks, organized by David Saad, was particularly notable. He succeeded in assembling an impressive list of speakers whose talks spanned essentially all of the major research issues in on-line learning. The workshop took place from 17 to 21 November, with the Newton Institute's purpose-designed building providing a superb setting.

This book resulted directly from the workshop, and comprises invited chapters written by each of the workshop speakers. It represents the first book to focus exclusively on the important topic of on-line learning in neural networks. On-line algorithms, in which training patterns are treated sequentially, and model parameters are updated after each presentation, have traditionally played a central role in many neural network models. Indeed, on-line gradient descent formed the basis of the first effective technique for training multi-layered networks through error back-progagation. It remains of great practical significance for training large networks using data sets comprising several million examples, such as those routinely used for optical character recognition.