[Linear Algebra] Four Views of Matrix Multiplication

1. Matrix Multiplications as Linear Combinations of Vector

• Inspired by the vector equation, we can view $A\textbf{x}$ as a linear combination of columns of the left matrix

e.g.

$\begin{bmatrix} 2 & 2 & 1 \\ 3 & 1 & 0 \\ 1 & 3 & 1 \\ \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} = A\textbf{x} = \begin{bmatrix} \textbf{a}_1 & \textbf{a}_2 & \textbf{a}_3 \\ \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} = \textbf{a}_1 x_1 + \textbf{a}_2 x_2 + \textbf{a}_3 x_3$

2. Matrix Multiplications as Column Combinations

• Linear combinations of columns
  • left matrix : bases
  • right matrix : coefficients

e.g.

$\begin{bmatrix} 2 & 2 & 1 \\ 3 & 1 & 0 \\ 1 & 3 & 1 \\ \end{bmatrix} \begin{bmatrix} 1 & 2 \\ 3 & 4 \\ 5 & 6 \end{bmatrix} = \begin{bmatrix} x_1 & y_1 \\ x_2 & y_2 \\ x_3 & y_3 \end{bmatrix} = \begin{bmatrix} \textbf{x} & \textbf{y} \\ \end{bmatrix}$

$\textbf{x} = \begin{bmatrix} 2 \\ 3 \\ 1 \\ \end{bmatrix} 1 + \begin{bmatrix} 2 \\ 1 \\ 3 \\ \end{bmatrix} 3 + \begin{bmatrix} 1 \\ 0 \\ 1 \\ \end{bmatrix} 5$

$\textbf{y} = \begin{bmatrix} 2 \\ 3 \\ 1 \\ \end{bmatrix} 2 + \begin{bmatrix} 2 \\ 1 \\ 3 \\ \end{bmatrix} 4 + \begin{bmatrix} 1 \\ 0 \\ 1 \\ \end{bmatrix} 6$

3. Matrix Multiplications as Row Combinations

• Linear combinations of rows of the right matrix
  • right matrix : bases
  • left matrix : coefficients

e.g.

$\begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ \end{bmatrix} \begin{bmatrix} 2 & 2 & 1 \\ 3 & 1 & 0 \\ 1 & 3 & 1 \\ \end{bmatrix} = \begin{bmatrix} x_1 & x_2 & x_3 \\ y_1 & y_2 & y_3 \\ \end{bmatrix} = \begin{bmatrix} \textbf{x}^T \\ \textbf{y}^T \\ \end{bmatrix}$

$\textbf{x}^T = \begin{bmatrix} 2 \\ 2 \\ 1 \\ \end{bmatrix}^T 1 + \begin{bmatrix} 3 \\ 1 \\ 0 \\ \end{bmatrix}^T 2 + \begin{bmatrix} 1 \\ 3 \\ 1 \\ \end{bmatrix}^T 3$

$\textbf{y}^T = \begin{bmatrix} 2 \\ 2 \\ 1 \\ \end{bmatrix}^T 4 + \begin{bmatrix} 3 \\ 1 \\ 0 \\ \end{bmatrix}^T 5 + \begin{bmatrix} 1 \\ 3 \\ 1 \\ \end{bmatrix}^T 6$

4. Matrix Multiplications as Sum of (Rank-1) Outer Products

• (Rank-1) outer product

  e.g.

  $\begin{bmatrix} 1 \\ 1 \\ 1\end{bmatrix} \begin{bmatrix} 1 & 2 & 3 \\ \end{bmatrix} = \begin{bmatrix} 1 & 2 & 3 \\ 1 & 2 & 3 \\ 1 & 2 & 3 \\ \end{bmatrix}$

• Sum of (Rank-1) outer products

  e.g.

  $\begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix} \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix} = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} \begin{bmatrix} 1 & 2 & 3 \end{bmatrix} + \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} \begin{bmatrix} 4 & 5 & 6 \end{bmatrix}$

• Sum of (Rank-1) outer products is widely used in machine learning

  • covariance matrix in multivariate Gaussian
  • gram matrix in style transfer