[Linear Algebra] Geometric Interpretation of Least Squares

Geometric Interpolation of Least Squares

• The vector $\textbf{b}$ is closer to $A \hat{\textbf{x}}$ than to $A \textbf{x}$ for other $\textbf{x}$
• To satisfy this, the vector $\textbf{b} - A \hat{\textbf{x}}$ should be orthogonal to $\textrm{Col} A$
  • This means $\textbf{b} - A \hat{\textbf{x}}$ should be orthogonal to any vector in $\textrm{Col} A$
• $\textbf{b} - A \hat{\textbf{x}} \perp x_1\textbf{a}_1 + x_2\textbf{a}_2 + \cdots + x_n\textbf{a}_n$ for any vector $\textbf{x}$
  • Or equivalently, $A^T (\textbf{b} - A \hat{\textbf{x}}) = \textbf{0}$