11 - Conjugate direction/gradient methods

Summary

The major impetus for the development of conjugate direction/gradient methods stems from the weakness of the steepest descent method (Chapter 10). Recall that while the search directions which are the negative of the gradient of the function being minimized can be computed rather easily, the convergence of the steepest descent method can be annoyingly slow. This is often exhibited by the zig-zag or oscillatory behavior of the iterates. To use an analogy, there is lot of talk with very little substance. The net force that drives the iterates towards the minimum becomes very weak as the problem becomes progressively ill-conditioned (see Remark 10.3.1). The reason for this undesirable behavior is largely a result of the absence of transitivity of the orthogonality of the successive search directions (Exercise 10.5). Consequently the iterates are caged up in a smaller (two) dimensional subspace and the method is unable to exploit the full n degrees of freedom that are available at our disposal. Conjugate direction method was designed to remedy this situation by requiring that the successive search directions are mutually A-Conjugate (Exercise 10.6). A-Conjugacy is a natural extension of the classical orthogonality. It can be shown that if a set of vectors are A-Conjugate, then they are also linearly independent. Thus, as the iteration proceeds conjugate direction/gradient method guarantees that the iterates minimize the given function in subspaces of increasing dimension. It is this expanding subspace property which is a hallmark of this method that guarantees convergence in almost n steps provided that the arithmetic is exact. Conjugate gradient (CG) method is a special class of conjugate direction (CD) method where the mutually A-Conjugate directions are recursively derived using the gradient of the function being minimized.