Question Updated
You need simultaneous zeros for $x' = y' = 0$.
From the second equation, we have $y = x$, sub back into first and we have:
$$x(1 - x) = 0 \implies x = 0, 1$$
Now find the two $y$ values, which are $y = 0, 1$.
So, we have two critical points as:
$$(x, y) = (0, 0), (1, 1)$$
Try those in each equation and verify that you get zero for each one.
Next, find the eigenvalues of the Jacobian matrix for each critical (fixed) point. We have:
$$J(x,y) = \begin{bmatrix} \frac{\partial x'}{\partial x} & \frac{\partial x'}{\partial y} \\ \frac{\partial y'}{\partial x} & \frac{\partial y'}{\partial y} \end{bmatrix} = \begin{bmatrix} -2x & 1 \\ -1 & 1 \end{bmatrix}$$
Do you see what the linearized system is now?
Now, evaluate the eigenvalues of $J(0,0)$ and $J(1,1)$.
For $J(0,0)$, we get $\lambda_1 = (-1)^{1/3},~ \lambda_2 = -(-1)^{2/3}$.
For $J(1,1)$, we get $\lambda_1 = \dfrac{1}{2}(-1 - \sqrt{5}),~ \lambda_2 = \dfrac{1}{2}(-1 + \sqrt{5})$.
Update 2
To find the linear approximation to the function $f(x)$ near a point $x^*$, we make use of Taylor’s theorem:
$$f(x) ≈ f(x^∗) + f′(x^∗)(x − x^∗).$$
There is a similar version of this theorem for functions depending on two variables.
Indeed,
$$F(x, y) ≈ F(x^∗, y^∗) + \dfrac{\partial F}{\partial x}(x^∗,y^∗)(x − x^∗) + \dfrac{\partial F}{\partial y} (x^∗, y^∗)(y − y^∗).$$
At a critical point, $F(x^∗, y^∗) = 0 = G(x^∗, y^∗)$. So the linear approximation is:
$$ \begin{bmatrix} x \\ y \end{bmatrix}' = \begin{bmatrix} \dfrac{\partial F}{\partial x} (x^∗, y^∗) & \dfrac{\partial F}{\partial y} (x^∗, y^∗) \\ \dfrac{\partial G}{\partial x} (x^∗, y^∗) & \dfrac{\partial G}{\partial y} (x^∗, y^∗) \end{bmatrix} \begin{bmatrix} x-x^* \\ y-y^* \end{bmatrix}$$
We can clean this up a little if we set $u = x − x^∗$ and $v = y − y^∗$. Then the linearization near $(x^∗, y^∗)$ becomes:
$$ \begin{bmatrix} u \\ v \end{bmatrix}' = \begin{bmatrix} \dfrac{\partial F}{\partial x} (x^∗, y^∗) & \dfrac{\partial F}{\partial y} (x^∗, y^∗) \\ \dfrac{\partial G}{\partial x} (x^∗, y^∗) & \dfrac{\partial G}{\partial y} (x^∗, y^∗) \end{bmatrix} \begin{bmatrix} u \\ v \end{bmatrix} = J\begin{bmatrix} u \\ v \end{bmatrix}$$
Notes: this is exactly what was derived above!
Notes: Do the calculations and make sure you get the same result, here are the intermediate calculations.
- $\dfrac{\partial F}{\partial x} = -2x$
- $\dfrac{\partial F}{\partial y} = 1$
- $\dfrac{\partial G}{\partial x} = -1$
- $\dfrac{\partial G}{\partial y} = -1$
- $(x^*, y^*) = (0, 0), (1, 1)$
Here are some Nonlinear Differential Equations, Section 3.2 on the Taylor approach for another perspective and for you to follow along with.