I wish to find 2 integers $a,b\ \ (a\neq b)$ so that $GM(a,b)$ and $RMS(a,b)$ are integers.
So I write down: $$\sqrt{ab}=c\ \ \rightarrow ab=c^2\\\sqrt{\frac{a^2+b^2}{2}}=d\ \ \rightarrow a^2+b^2=2d^2$$ By lots of trial and error I think there aren't any solutions, but I didn't find a way to prove.
Is that true that there aren't solutions? And how do I prove it?
If $a$ and $b$ are equal, then of course the above are satisfied.
$a^2+b^2 + 2ab = (a+b)^2 = 2(d^2+c^2)$ and $(a-b)^2 = 2(d^2-c^2)$. Hence, both $2(d^2-c^2)$ and $2(d^2+c^2)$ are perfect squares.
Let $e^2 = 2(d^2-c^2)$ and $f^2=2(d^2+c^2)$. Then, $ e^4 - f^4 = (e^2-f^2)(e^2+f^2) = 16d^2c^2$. In other words, $$ e^4 = f^4 + (4dc)^2 $$ This is a solution to the equation $x^4-y^4 = z^2$, with $x=e,y=f,z=4dc$.
I claim that $x^4-y^4=z^2$ has no non-trivial solutions.
Suppose that $x,y,z$ is the smallest (the smallest means that $x^2+y^2$ is the minimized quantity) non-trivial solution of $x^4-y^4=z^2$ (which exists if there is a non-trivial solution).
Then $z,y^2,x^2$ are Pythagorean triplets. Furthermore, they are primitive (which means that there is no common factor they share), otherwise we can get a smaller solution. Then $z,y,x$ can be written in the form (assuming $z$ is even)$ z = 2pq, y^2 = p^2-q^2, x^2 = p^2+q^2$ ($p$ and $q$ of different parity and coprime, otherwise $z,y,x$ would have common factor $2$). Plug this back in, and you get $p^4 - q^4 = (xy)^2$. Hence, we can remove all the factors of $2$ from $z$, and assume that it is odd, in which case $y^2 = 2pq$ and $z = p^2-q^2$.
If $z$ is odd, then $y$ is even (as $y^2$ must then be of them form $2pq$). Now, since $y^2=2pq$, we can write $q=2u^2$ and $p=v^2$, because $p$ and $q$ were assumed coprime.
Now, since $p^2+q^2=x^2$, we write $p=r^2-s^2,q=2rs,x=r^2+s^2$ for $r,s$ coprime . Hence, we get that $2u^2=2rs$, so we can write $r=g^2$ and $s=h^2$.
Finally, $$ p=v^2=r^2-s^2 = g^4-h^4 \implies v^2 = g^4-h^4 $$
Note that: $$ g^2+h^2=r+s<(r+s)(rs)(r−s)=\frac{1}{2}(r^2-s^2)(2rs)=\frac{pq}{2}=\frac{y^2}4 \leq y^2 < x^2+y^2 $$
This gives a contradiction,since we were assuming that the solution minimized $x^2+y^2$. Hence, the above problem (and your problem) have no non-trivial solutions.
Hence the solutions to your problem are only when $a=b$, so that the values of $c$ and $d$ are also trivial.
Finally, I would like to point out that $x^4+y^4 = z^4 \implies z^4-y^4=(x^2)^2$, which we showed has no solutions. Hence, we just proved Fermat's last theorem for the case $n=4$ as well.
