An equilateral triangle exists with vertices $(0,0), (a,11)$ and $(b,37)$. Using the distance formula three times, I eventually arrive to:
$$a^2 + 121 = b^2 + 1369 = a^2 + b^2 - 2ab + 676$$
The only problem is that since none of the equations are actually equal to anything, I can't figure it out. How can I solve this?
Denote the three vertices by $O$, $A$, $B$, and let $M=(m,24)$, $m>0$, be the midpoint of $AB$. Then $h:=|OM|$ is the height of the triangle and $s={2\over\sqrt{3}} h$ its sidelength. If $\delta$ is the angle between $OM$ and the positive $y$-axis we have $${m\over h}=\sin\delta={13\over s/2}={13\sqrt{3}\over h}\ ,$$ so that $m=13\sqrt{3}$. This leads to $h=\sqrt{m^2+24^2}=19\sqrt{3}$, hence $s=38$. Finally $$a=\sqrt{s^2-11^2}=21\sqrt{3},\qquad b=\sqrt{s^2-37^2}=5\sqrt{3}\ .$$ A second solution is obtained by reflecting this triangle with respect to the $y$-axis.
You actually have two equations: for instance, \begin{align} a^2+121&=b^2+1369\\ b^2+1369&=a^2+b^2-2ab+676. \end{align} It is easy to see that $b=0$ gives no solution: the second equation becomes $121=676$. So we assume $b\ne0$. We rewrite the equations as \begin{align} a^2-b^2&=1248\\ a^2-2ab&=693. \end{align} If you subtract both equations, you get $$ -b^2+2ab=555. $$ Thus, $$ a=\frac{555+b^2}{2b}.$$ If we plug this into $a^2-b^2=1248$, $$ 1248=\frac{555^2+b^4+1110b^2}{4b^2}-b^2 =\frac{555^2-3b^4+1110b^2}{4b^2}, $$ so $$ 3b^4+3882b^2-555^2=0. $$ Now \begin{align} b^2&=\frac{-3882\pm\sqrt{3882^2+4\times3\times555^2}}{6} =\frac{-3882\pm\sqrt{2^4\times 3^2\times 19^4}}{6}\\ &=\frac{-3882\pm2^2\times3\times 19^2}{6} =647\pm722, \end{align} so $b^2=75$ (the negative solution is not acceptable because it won't give $b$ real), and then $b=\pm5\sqrt3$.
Correspondingly, $$ a=\pm\sqrt{1248+b^2}=\pm\sqrt{1248+75}=\pm\sqrt{1323}=\pm21\sqrt3. $$
