I would like to find all solutions to
$$b-a\sqrt{1+a^2+b^2}=a^2(ab-\sqrt{1+a^2+b^2})$$ $$a-b\sqrt{1+a^2+b^2}=b^2(ab-\sqrt{1+a^2+b^2})$$
I found some solutions. For example, $a = 1, b = \pm i$ and $b = 1 , a = \pm i$. How can I find all solutions?
I would like to find all solutions to
$$b-a\sqrt{1+a^2+b^2}=a^2(ab-\sqrt{1+a^2+b^2})$$ $$a-b\sqrt{1+a^2+b^2}=b^2(ab-\sqrt{1+a^2+b^2})$$
I found some solutions. For example, $a = 1, b = \pm i$ and $b = 1 , a = \pm i$. How can I find all solutions?
One technique with this kind of problem is to isolate the square root, and then square the equation (which brings in possible extra roots from the negative value of the square root).
[Now corrected for error pointed out by Anush in copying over the equations]
Beginning that process here we find that the first equation becomes: $$b(1-a^3)=(a-a^2)\sqrt{1+a^2+b^2}$$ and notice that the factor $a-1$ will cancel, provided that $a\neq 1$ so that$$b(1+a+a^2)=a\sqrt{1+a^2+b^2}$$
Similarly, from the second equation, if $b\neq 1$ we have $$a(1+b+b^2)=b\sqrt{1+a^2+b^2}$$
Now we can eliminate the square root by subrtacting $a\times$ the second equation from the $b \times$ first to obtain $$(b^2-a^2)+ab(b-a)=(b-a)(a+b+ab)=0$$
So that either $ab+a+b=0$ or $a=b$
This is substantial progress.
If $a=b$ we note the solution $a=b=0$, and assume that $a,b\neq 0, 1$. The two equations are the same and we need to solve $$(1+a+a^2)=\sqrt{1+2a^2}$$
If we square this we obtain $$1+2a+3a^2+2a^3+a^4=1+2a^2$$ which becomes $$2a+a^2+2a^3+a^4=0$$ or $$a(2+a)(a^2+1)=0$$
$a=0$ we know about. $a=-2, b=-2$ is another solution. $a=b=\pm i$ works in the original equation for $a=b=i$ but not for $a=b=-i$.
If now $a+b+ab=0$, the term $b(1+a+a^2)=b+ab+a^2b=-a + a^2b=a(ab-1)$ so we have
$a(ab-1)=a\sqrt{a^2+b^2+1}$ and if $a\neq 0$ we can cancel and square to obtain $$(ab-1)^2=a^2+b^2+1$$ Now we can write $ab=-(a+b)$ so that the equation becomes $$a^2+b^2+1+(2ab+2a+2b)=a^2+b^2+1$$ The term in brackets is zero, so the equation is an identity and gives no additional constraint.
Note that the condition $ab+a+b=0$ is equivalent to $(a+1)(b+1)=1$, and if $b\neq -1$ this is also $a=\cfrac {-b}{b+1}$
Some checking is required to identify which solutions get the right sign for the square root - assuming the solutions are real numbers (and complex solutions have been canvassed). We know we have $a^2+b^2+1=(ab-1)^2$ so that $\sqrt {a^2+b^2+1}=|ab-1|$, so for a solution we need $ab\gt 1$. This in turn means $\frac{-b^2}{b+1} \gt 1$.
If $b+1\gt 0$ this reduces to $-b^2\gt b+1\gt 0$ which is impossible. So $b+1\lt 0$ (ignoring $b=-1$) and $$-b^2\lt b+1$$ or $b^2+b+1\gt 0$ which is equivalent to $(2b+1)^2+3\gt 0$, which is always true.
So we have the condition $b\lt -1$ and from the form $(a+1)(b+1)=1$ we see that this also implies $a\lt-1$
The first equation becomes $$b(1-a^3)=a(1-a)\sqrt{1+a^2+b^2},$$ the second $$a(1-b^3)=b(1-b)\sqrt{1+a^2+b^2}.$$
If $a=0$, then $b=0$.
Clearly $a=b=1$ is a solution. Let $a=1$ and $b\neq1$. Then from the second equation $$1+b+b^2=b\sqrt{2+b^2}$$ we derive the solutions $b=\pm i$.
Let $a$ and $b$ not equal 0 or 1. Then we have $$b(1+a+a^2)=a\sqrt{1+a^2+b^2},$$ and $$a(1+b+b^2)=b\sqrt{1+a^2+b^2}.$$
Consider now $1+a+a^2=0\iff a=\frac{-1\pm i\sqrt{3}}{2}$. In this case $1+a^2+b^2$ must be 0, this gives together with $1+a+a^2=0$ the condition $b^2=-a$, from where we get four more solutions.
Now lets divide one equation by the other. An easy compution yields $$(b-a)(a+b+ab)=0.$$ Now if you multiply the equations, you'll get $$ab(1+ab)(a+b+ab)=0\iff (1+ab)(a+b+ab)=0.$$ So either $a+b=-ab$ or $a=b$ and $ab=-1$, that is $a=b=i$.
So we're done. (Remind that if $(a,b)$ is a solution, so is $(b,a)$.
Michael