understanding partial derivatives in backpropagation algorithm

Question

This question is about derivatives. It is okay if you don't know backpropagation.
I was following this guide to understand backpropagation algorithm (equation 36, 37 and 38).

Here is a simplified form of the problem. Let $$z_1 = w_{11}*x_1 + w_{21}*x_2 + w_{31}*x_3$$ $$z_2 = w_{12}*x_1 + w_{22}*x_2 + w_{32}*x_3$$ And, $$a_1 = F(z_1)$$ $$a_2 = F(z_2)$$ We also have a function $C(a_1,a_2)$
Now they applied multivariable chain rule to find, $$\frac{\partial C}{\partial z_1} = \frac{\partial C}{\partial a_1}\frac{\partial a_1}{\partial z_1} + \frac{\partial C}{\partial a_2}\frac{\partial a_2}{\partial z_1}$$
Then they stated that $$\frac{\partial a_2}{\partial z_1} = 0 $$ I don't understand why is this term equal to 0?
According to what i learnt from this answer, i think it should be $$\frac{\partial a_2}{\partial z_1} = F'(z_2)\frac{\partial z_2}{\partial z_1} = F'(z_2)(\frac{\partial z_2}{\partial x_1}\frac{\partial x_1}{\partial z_1}+\frac{\partial z_2}{\partial x_2}\frac{\partial x_2}{\partial z_1}+\frac{\partial z_2}{\partial x_3}\frac{\partial x_3}{\partial z_1})$$ $$\frac{\partial a_2}{\partial z_1} = F'(z_2)(\frac{w_{12}}{w_{11}} + \frac{w_{22}}{w_{21}} + \frac{w_{32}}{w_{31}})$$

Please tell me where i am wrong

Answer 1

Now I think I understand. Based on the tutorial you are referencing, the $x_i$'s are not your variables, they are your data. Instead, the variables $z_1$ and $z_2$ are functions of the the weights $w_{11}, w_{12}, \ldots$.

Here is an updated diagram that may help. You are concerned with the output of the node containing $z_2$. Your function $F$ is only being applied to the output from $z_2$. There is no connection that takes you directly from $z_1$ to $z_2$. That is, the derivative is 0.

Note that this is a picture of the variable dependence not your neural network. In the network diagram, the $w$'s are the edges.

Your equations now look like

$$ \begin{align} a_2 &= F(z_2)\\ z_2 &= f(w_{12},w_{22},w_{32})\\ z_1 &= f(w_{11},w_{21},w_{31}) \end{align} $$

So it is clear that

$$ \frac{\partial a_2}{\partial z_1} = 0 $$

Answer 2

This is partial derivatives with non independent variables. $a_2$ and $z_1 $ are not independent since $z_1$ and $z_2$ are composed of the same variables. That is to say, a change is $z_1$ will cause a change in $z_2$ in turn causing $a_2$ to change.

I think the problem here is that some information is missing. You have too many variables and not enough equations that relate them. I'll illustrate by example:

Consider the following equations $$ w=x^2+y^2 +z^2 \\ z=x^2 + y^2 $$

The number of variables here is 4 (w,x,y,z) and two equations. This means only two variables can be chosen independently. If you are asked to find $\frac{ \partial w}{\partial z} $ and not told anything else, you cant compute. You need to be told which variables you are going to hold constant while calculating partial derivatives. If the question is phrased as find $\frac{ \partial w}{\partial z} $ while keeping x constant, then you can calculate.

You need to be given enough information so that the number of independent variables is just 1. In fact, I suspect that the material you are referring is trying to tell you to hold $a_2$ as a constant.

Here's a pdf from MIT ocw that might help https://ocw.mit.edu/courses/mathematics/18-02sc-multivariable-calculus-fall-2010/2.-partial-derivatives/part-c-lagrange-multipliers-and-constrained-differentials/session-43-clearer-notation/MIT18_02SC_MNotes_n3.pdf