10 - Optimization: steepest descent method

Summary

In Chapters 5 and 7 the least squares problem – minimization of the residual norm, f (x) = ∥r(x)∥ where r(x) = (z − Hx) in (5.1.11) and (7.1.3) with respect to the state variable x is formulated. There are essentially two mathematically equivalent approaches to this minimization. In the first, compute the gradient ∇ f (x) and obtain (the minimizer) x by solving ∇ f (x) = 0. We then check if the Hessian ∇²f (x) is positive definite to guarantee that x is indeed a local minimum. In the linear least squares problem in Chapter 5, f (x) is a quadratic function x and hence ∇ f (x) = 0 leads to the solution of a linear system of the type Ax = b with A a symmetric and positive definite matrix (refer to (5.1.17)) which can be solved by the methods described in Chapter 9. In the nonlinear least squares problem, f (x) may be highly nonlinear (far beyond the quadratic nonlinearity). In this case, we can compute x by solving a nonlinear algebraic system given by ∇ f (x) = 0, and then checking for the positive definiteness of the Hessian ∇²f(x). Alternatively, we can approximate f(x) locally around a current operating point, say, x_c by a quadratic form Q(y) (using either the first-order or the second-order method described in Chapter 7) where y = (x – x_c).