6 - A geometric view: projection and invariance

Summary

In this chapter we revisit the linear least squares estimation problem and solve it using the method of orthogonal projection. This geometric view is quite fundamental and has guided the development and extension of least squares solutions in several directions. In Section 6.1 we describe the basic principles of orthogonal projections, namely, projecting a vector z onto a single vector h. In Section 6.2, we discuss the extension of this idea of projecting a given vector z onto the subspace spanned by the columns of the measurement matrix H ∈ ℝ^m×n. An interesting outcome of this exercise is that the set of linear equations defining the optimal estimate by this geometric method are identical to those derived from the method of normal equations. This invariance of the least squares solution with respect to the methods underscores the importance of this class of solutions. Section 6.3 develops the geometric equivalent of the weighted or generalized linear least squares problem. It is shown that the optimal solution is given by an oblique projection as opposed to an orthogonal projection. In Section 6.4 we derive conditions for the invariance of least squares solutions under linear transformations of both the model space ℝⁿ and the observation space ℝ^m. It turns out invariance is achievable within the framework of generalized or weighted least squares formulation.

Orthogonal projection: basic idea

Let h = (h₁, h₂, …, h_m)^T ∈ ℝ^m be the given vector representing the measurement system, and let z = (z₁, z₂, …, z_m)^T ∈ ℝ^m be a set of m observations, where it is assumed that z is not a multiple of h. Refer to Figure 6.1.1.