Lec 7: Backpropagation

• Backpropagation 是一種比較有效率的 gradient descent

Chain Rule

• $y = g(x), z = h(y)$

  • $\dfrac{\text{d}z}{\text{d}x} = \dfrac{\text{d}z}{\text{d}y} \dfrac{\text{d}y}{\text{d}x}$

• $x=g(s), y=h(s), z=k(x,y)$

  • $\dfrac{\text{d}z}{\text{d}s} = \dfrac{\text{d}x}{\text{d}s} \dfrac{\partial z}{\partial x} + \dfrac{\text{d}y}{\text{d}s}\dfrac{\partial z}{\partial y}$

Backpropagation

$L(\theta) = \sum_{n=1}^N C^n(\theta)$ so that $\dfrac{\partial L(\theta)}{\partial w} = \sum_{n=1}^N \dfrac{\partial C^n(\theta)}{\partial w}$ 所以現在只需要 focus 在如何計算某筆 training data 的 loss

• Forward pass
  • compute $\dfrac{\partial z}{\partial w}$ for all params $w$
• Backward pass
  • compute $\dfrac{\partial C}{\partial z}$ for all activation function input $z$

Forward pass

$\dfrac{\partial z}{\partial w}$ 其實就是他的 input

Backward pass

$\dfrac{\partial C}{\partial z} = \dfrac{\partial a}{\partial z}\dfrac{\partial C}{\partial a}$

• 以 sigmoid 而言，$\partial a / \partial z = \sigma(z)(1-\sigma(z))$

而根據 chain rule，$\partial C / \partial a = \sum_\limits{z^{[l+1]}\in Z^{[l+1]}} \dfrac{\partial z^{[l+1]}}{\partial a}\dfrac{\partial C}{\partial z^{[l+1]}}$ 又 $\partial z_k^{[l+1]}/\partial a = w_k^{[l+1]}$

因此 $\dfrac{\partial C}{\partial z} = \dfrac{\partial a}{\partial z} \sum_\limits{z^{[l+1]}\in Z^{[l+1]}} w_k^{[l+1]}\dfrac{\partial C}{\partial z_k^{[l+1]}}$

• $\partial a / \partial z$ 其實就是該層 activation function 的微分 (在 z 值 的導數 $\sigma'(z)$，而 z 在 forward pass 時已經知道，因此這一項是常數)
• $\partial C / \partial z_k^{[l+1]}$ 就是下一層某個 neuron 的 $\partial C / \partial z$

所以要求某層的 $\partial C / \partial z$ 就只要從最後一層算回來就好，故名 backward pass

Backpropagation - Summary

$\partial C / \partial w = \dfrac{\partial z}{\partial w}\times \dfrac{\partial C}{\partial z}$

只要是跟第 k 個 input 相連的 w 的 forward pass

• $\dfrac{\partial z}{\partial w_k} = a_k^{[l-1]}$
  • 就是跟著他的 input

backward pass

• $\dfrac{\partial C}{\partial z} = \sigma'(z) \sum_\limits{z_k^{[l+1]}\in Z^{[l+1]}} w_k^{[l+1]}\dfrac{\partial C}{\partial z_k^{[l+1]}}$