As the title states, I want to find the derivative of
$$\frac{1}{N}\sum_i (x_i - \mu)^2$$ w.r.t $x_i$ (note that $\mu$ is also another function of $x_i$, of course). I've tried solving it and got the following result
$$\frac{2(N - 1)}{N^2}\sum_i (x_i - \mu)$$
Is this right? Am I doing something wrong?
I think it's easier to understand by expanding the summation.
$$\frac\partial{\partial x_i}\frac 1N\sum_j^N (x_j-\mu)^2$$ $$= \frac 1N\sum_i^N \frac\partial{\partial x_i}(x_i-\mu)^2$$ If we expand the summation, we get $$= \frac 1N \Biggl[\frac\partial{\partial x_i}(x_0-\mu)^2 + \cdots + \frac\partial{\partial x_i}(x_i-\mu)^2 + \cdots + \frac\partial{\partial x_i}(x_N-\mu)^2\Biggl]$$ All other derivative in the bracket is zero except $\frac\partial{\partial x_i}, (x_i-\mu)^2=2(x_i - \mu)$; hence $$= \frac 2N (x_i - \mu)$$
Note that $\partial\mu/\partial x_i=1/N$. If $j\ne i$ then $\partial x_j/\partial x_i=0$, while if $j=i$ then $\partial x_j\partial x_i=1$. So $$\begin{align}\frac\partial{\partial x_i}\frac 1N\sum_j(x_j-\mu)^2 &=\frac 1N\sum_j2(x_j-\mu)\frac{\partial (x_j-\mu)}{\partial x_i} \\&=-\frac1{N^2}\sum_j2(x_j-\mu)+\frac 1N2(x_i-\mu) \\&=\frac{2(x_i-\mu)}{N},\end{align}$$since $\sum(x_j-\mu)=0$.
Or, maybe less confusing, start with $\sigma^2=\Bbb E X^2-\mu^2$:
$$\frac\partial{\partial x_i}\left(\frac 1N\sum_j x_j^2-\mu^2\right) =\frac 2Nx_i-2\mu\frac{\partial\mu}{\partial x_i}.$$
It's nos right. From your derivation I can see that you are derivating every $x_i$, but you want to derivate respect to just one. Maybe the notation is what is causing the problem. The correct answer is this:
$$\dfrac{\partial S^2}{\partial x_i}=\dfrac{1}{N}\left[2(x_1-\mu)\dfrac{-1}{N}+\cdots+2(x_i-\mu)\dfrac{N-1}{N}+\cdots+2(x_N-\mu)\dfrac{-1}{N}\right]$$ $$=\dfrac{2}{N^2}\left[(x_i-\mu)(N-1)-\sum_{k\neq i}(x_k-\mu)\right]$$ $$=\dfrac{2}{N^2}\left[x_iN-x_i-\sum_kx_k+\mu-\sum_kx_k+N\mu+x_i-\mu\right]$$ $$=\dfrac{2}{N^2}\left[Nx_i-\sum_kx_k\right].$$
