Aspired by sir J.G., I now have a rough idea how to understand the problem.
$d\mathbf{x}$ is a vector, it has a direction and magnitude.
$d\mathbf{y}$ is another vector, is has another direction and mangnitude.
First we what calulate $d\mathbf{y}/d\mathbf{x}$, suppose
$$\frac{d\mathbf{y}}{d\mathbf{x}}=\mathbf{M}$$
thus
$$d\mathbf{y}=\mathbf{M}d\mathbf{x}$$
we can see that $\mathbf{M}$ is a mapping from $d\mathbf{x}$ to $d\mathbf{y}$. It leads to the rotation and deformation on the $d\mathbf{x}$.
Then we want calculate the $d\mathbf{x}/d\mathbf{y}$. Suppose
$$\frac{d\mathbf{x}}{d\mathbf{y}}=\mathbf{Q}$$
we can see that $\mathbf{Q}$ is a mapping from $d\mathbf{y}$ to $d\mathbf{x}$. It leads to the rotation and deformation on the $d\mathbf{y}$.
thus
$$d\mathbf{x}=\mathbf{Q}d\mathbf{y} = \mathbf{QM}d\mathbf{x}$$
$$ \mathbf{QM} = \mathbf{I}$$
$$ \mathbf{Q} = \mathbf{M}^{-1}$$
thus $\mathbf{Q}$ is a mapping can undo the effect of $\mathbf{M}$, that is the invert matrix of $\mathbf{M}$.
So the "inverse derivate of vector" should be a invert matrix $\mathbf{M}^{-1}$ but not the matrix organized by invert component $(M_{ij}^{-1})$.