stable and unstable limit cycles

Question

Given a system $$\dot x = -y + xF(r), \ \dot y = x + y F(r) $$ where $F(r):= r^4-3r^2+1$.

I need to find out the stable and unstable limit cycle for the given system.

My Approach

I used the polar coordinates to reduce the system as $$\dot r = r(r^4-3r^2+1), \ \dot \theta =1$$

The reduced system has a unique critical point at origin since $\dot \theta \neq 0$.

So to find the stable and unstable limit points all I need to check is the sign of $\dot r$ in the annulus. I am not able to follow up from here.

Answer 1

Letting $$ r^4-3r^2+1=0$$ gives two positive roots $$ r_{1,2}=\sqrt{\frac{3\pm\sqrt{5}}{2}}. $$ Then the equation becomes $$ r'=r(r^2-r_1^2)(r^2-r_2^2). $$ and $$ x^2+y^2=r_{1,2}^2$$ are two solution curves. Consider the region $$ R_k=\{r: r_k-\epsilon <r<r_k+\epsilon\}$$ for small $\epsilon>0$. Then on $r=r_1-\epsilon$, $r'<0$ and on $r=r_1+\epsilon$, $r'>0$ and hence $r=r_1$ is an unstable limit cycle. Similarly $r=r_2$ is another unstable limit cycle in $R_2$ too.