Back propagation equation proof

Question

I am trying to prove this equation (from the backpropagation equations in AI).

$$\frac{\partial C}{\partial b_j^l} = \delta_j^l$$

C is the cost function: $C = \frac{1}{2}||y - a^L||^2$

Where the output of layer l and neuron j is express like so $a^l_j=σ(∑_kw^l_{jk}a^{l−1}_k+b^l_j)$

I am suppose to use this assertion to do the demonstration: $\delta_j^L = \frac{\partial C}{\partial z_j^L}$

So far, here is what I have tried:

$$\frac{\partial C}{\partial z_j^L} = \sum_k \frac{\partial C}{\partial z_j^L} \frac{\partial b_j^l}{\partial b_j^L} $$ (I am using the chain rule to have a sum)

<=> $$\frac{\partial C}{\partial z_j^L} = \sum_k \frac{\partial C}{\partial b_j^l} \frac{\partial b_j^l}{\partial z_j^L} $$

So I guess I have to prove, that $\frac{\partial b_j^l}{\partial z_j^L}$ equals 1.

But I don't have any ideas how to prove it.

Thanks for your help

N.B I am following this course => http://neuralnetworksanddeeplearning.com/chap2.html where the first two equations of the BackPropagation equations are already proved, and the 2 others should be proved the same way (using the chain rule)

Answer 1

By definition, $$ \frac{\partial C}{\partial z_j^\ell} = \delta_j^\ell $$ $$ z^\ell_k = \sum_i w_{ki}^\ell a^{\ell -1}_j + b_k^{\ell} $$ the latter of which means that $$ \frac{\partial z^\ell_k}{\partial b_\alpha^\ell} = \Delta^\ell_{k\alpha}=\begin{cases} 1 & \text{if } k=\alpha \\ 0 & \text{otherwise} \end{cases} $$ We want to show that $$ \frac{\partial C}{\partial b_j^\ell} = \delta_j^\ell $$ Since $C$ depends on $b_\beta$ through $z^\beta$, we can use the chain rule: \begin{align} \frac{\partial C}{\partial b_j^\ell} &= \sum_\zeta \frac{\partial C}{\partial z_\zeta^\ell} \frac{\partial z_\zeta^\ell}{\partial b_j^\ell} \\ &= \sum_\zeta \delta^\ell_\zeta \Delta^\ell_{\zeta j} \\ &= \delta^\ell_j \end{align} where the Kronecker delta $\Delta_{\zeta j}^\ell$ collapses out the sum except for the $j$th term.