[Linear Algebra] Linear Transformation in Neural Networks

Linear Transformation in Neural Networks

• Fully-connected layer (linear layer, without bias)

  $\textbf{y} = A\textbf{x}$

  $A \in \mathbb{R}^{m \times n}$ : weight matrix of fully-connected layer
  $\textbf{x} \in \mathbb{R}^{n}$ : input vector
  $\textbf{y} \in \mathbb{R}^{m}$ : output vector

Affine Layer in Neural Networks

• Fully-connected layers usually have involve a bias term
• That is why we call it an affine layer, but not a linear layer

  $\textbf{y} = A\textbf{x} + \textbf{b}$

  $A \in \mathbb{R}^{m \times n}$ : weight matrix of fully-connected layer
  $\textbf{b} \in \mathbb{R}^{m}$ : bias vector of fully-connected layer
  $\textbf{x} \in \mathbb{R}^{n}$ : input vector
  $\textbf{y} \in \mathbb{R}^{m}$ : output vector