I have been given the following system to solve: $$(x^2+1)(y^2+1) = 12xy$$ $$(x+1)^2(y+1)^2=30xy$$
I noticed that the system is symmetric, but any of the methods I know to solve symmetric systems doesn't seem right. I have also tried writing the second equation as $$((x^2+1)+2x)((y^2+1)+2y) = (x^2+1)(y^2+1)+2y(x^2+1)+2x(y^2+1)+4xy = 30xy$$, but this didn't get me anywhere. I would appreciate any hints towards the right solution.
Divide both sides of both equations by $xy$ and let $u = x+\frac1x + 1$, $y = y+\frac1y + 1$. We have
$$\begin{cases}(u-1)(v-1) &= 12\\ (u+1)(v+1) &= 30\end{cases} \implies \left\{\begin{align} uv &= \frac12(30+12)-1 = 20\\ u + v &= \frac12(30-12) = 9 \end{align}\right. $$ This implies $u,v$ are roots of $\lambda^2 - 9\lambda + 20 = (\lambda - 4)(\lambda - 5)$.
Up to permutation of $x, y$, we have $$ \begin{cases} x + \frac1x = u - 1 = 3\\ y + \frac1y = v - 1 = 4 \end{cases} \iff \begin{cases} x^2 - 3x + 1 = 0\\ y^2 - 4y + 1 = 0 \end{cases} $$ This leads to 8 solutions of the problem (see illustrations below): $$(x, y) = \left(\frac{3 \pm \sqrt{5}}{2},2\pm \sqrt{3}\right) \text{ or } \left(2\pm \sqrt{3},\frac{3 \pm \sqrt{5}}{2}\right) $$
$$ \left\{ \begin{array}{c} \left(x^2+1 \right)\left(y^2+1 \right)=12xy \\ \left(x+1 \right)^2\left(y+1 \right)^2=30xy \end{array} \right. $$
$$ \iff \left\{ \begin{array}{c} \left(x^2+1 \right)\left(y^2+1 \right)=12xy \\ \left(x^2+1+2x \right)\left(y^2+1+2y \right)=30xy \end{array} \right. $$
$$ \iff \left\{ \begin{array}{c} \left(x+\frac{1}{x} \right)\left(y+\frac{1}{y} \right)=12 \\ \left(x+\frac{1}{x}+2 \right)\left(y+\frac{1}{y}+2 \right)=30 \end{array} \right. $$
Put $A=x+\frac{1}{x}$ and $B=y+\frac{1}{y}$, we have
$$ \left\{ \begin{array}{c} AB=12 \\ \left(A+2 \right)\left(B+2 \right)=30 \end{array} \right. $$
Solving the equations, we have $A=3, B=4$ or $A=4, B=3$.
When $A=3, B=4$ , we have
$$ \left\{ \begin{array}{c} x+\frac{1}{x}=3 \\ y+\frac{1}{y}=4 \end{array} \right. $$
$$ \left\{ \begin{array}{c} x=\frac{3\pm\sqrt{5}}{2}\\ y=2\pm\sqrt{3} \end{array} \right. $$
When $A=4, B=3$ , we have
$$ \left\{ \begin{array}{c} x+\frac{1}{x}=4 \\ y+\frac{1}{y}=3 \end{array} \right. $$
$$ \left\{ \begin{array}{c} x=2\pm\sqrt{3}\\ y=\frac{3\pm\sqrt{5}}{2} \end{array} \right. $$
Hint
From a formal point of view, you face an octic polynomial.
Use the first equation and solve for $y$ $$y_\pm=\frac{6x\pm\sqrt{-x^4+34 x^2-1}}{x^2+1}$$ Using $y_+$, plug it in the second, group terms and factor; you should have $$12 x \left(x^2-4 x+1\right) \left(x^2-3 x+1\right)+$$ $$2 \left(x^2-4 x+1\right) \left(x^2-3 x+1\right)\sqrt{-x^4+34 x^2-1}=0$$ that is to say $$\left(x^2-4 x+1\right) \left(x^2-3 x+1\right)(6x+\sqrt{-x^4+34 x^2-1})=0$$
Now, it is simple.
Do the same using $y_-$.
