The way I learned it, when determining the stability of fixed points in a non-linear two-dimensional dynamical system of the form $$ \dot{x} = f(x,y), \\ \dot{y} = g(x,y), $$
after determining the positions of all fixed points, I use the Jacobian matrix at those points to determine their stability, i.e. (the way I understood) we reduced the system at this point to a homogenous linear system (which we can easily work with). I would love understand why this is allowed and how it works.
The idea behind understanding the nature of equilibria of a dynamical system via linearization is that, at least locally, the system behaves like its approximation at the first order, hence like its linearization.
We know that if $(\bar{x},\bar{y})$ is an equilibrium it holds $(f(\bar{x},\bar{y}),g(\bar{x},\bar{y}))=(0,0)$ so the Taylor expansion of $F(x,y) := (f(x,y),g(x,y))$ about the point $(\bar{x},\bar{y})$ is the following: $F((\bar{x},\bar{y})) + JF(\bar{x},\bar{y})(x-\bar{x},y-\bar{y}) + O(||(x,y)||^2)$ i.e. the linearized system is the following:
$ \begin{bmatrix} \dot{x} \\ \dot{y} \end{bmatrix} \approx \begin{bmatrix} \frac{\partial f}{\partial x}(\bar{x},\bar{y})&\frac{\partial f}{\partial y}(\bar{x},\bar{y}) \\ \frac{\partial g}{\partial x}(\bar{x},\bar{y}) & \frac{\partial g}{\partial y}(\bar{x},\bar{y}) \end{bmatrix} \begin{bmatrix} x-\bar{x} \\ y-\bar{y} \end{bmatrix} $
So now since in proximity of the stationary point the system behaves like the linearized one. Hence we know that if the analysis of this simpler system tells us that the point is stable/unstable for this system, then it preserves its nature even for the more complex non linear system.
So this is the idea behind linearization, then in order to make your analysis about the stability of equilibria you have to refer to Hartman–Grobman theorem (known as Linearization theorem) as BAYMAX said.
Yes, in general this works for the case of hyperbolic fixed point, this has a name known as Hartman-Grobman theorem.
The same was also asked - see here.
The theorem is here -
Let $x \in \Bbb{R}^n$, consider the non-linear system $\dot{\vec{x}}= f(\vec{x})$ with the flow $\phi_{t}$ and the linear system $\dot{x} =Ax$ where $A$ is the Jacobian of $f$ evaluated at $x = x^*$(where $x^*$ is the hyperbolic fixed point). Let $U,V$ contain the fixed point $x^*$, then there exists a homeomorphism $H:U \rightarrow V$ such that for all initial points$x_{0} \in U$ and all time $t$, we have $H \circ \phi_{t}(\vec{x_{0}}) = e^{At} H(\vec{x_{0}})$.
For the proof of Hartman-Grobman theorem, you can refer here.
