$a$ and $b$ are distinct positive integers such that $\frac{a+b}{2}$, $\sqrt{ab}$, and $\frac{2}{\frac{1}{a}+\frac{1}{b}}$ are integers.
Find the smallest possible value of $|a-b|$.
My work led me to find that one possible pair is $a=90, b=10$, but $80$ is not the minimum value of the difference.
Your idea of having $a=n^2b$ seems good, so we can try $n=2$. Then $a=4b$ and we want $\frac {a+b}2=\frac {5b}2, \sqrt{ab}=2b,$ and $\frac {2ab}{a+b}=\frac {8b}5$ to be integers, which we can satisfy with $b=10, a=40, |a-b|=30$
To prove minimality, WOLOG let $a \ge b$. Assume $b$ does not divide $a$. Let $p=\gcd(a,b)$. To make the GM integral, we must have $b=m^2p, a=n^2p$, with $m,n \gt 1$ and $m,n,p$ pairwise coprime. The HM is $\frac {2ab}{a+b}=\frac{2m^2n^2p^2}{p(m^2+n^2)}$ To make this an integer, we must have $(m^2+n^2)|p,$ so $p \ge 13$ and $a-b \ge (9-4)13=65$, worse than the $30$ case we have found.
As $b$ divides $a$, the GM forces $a=n^2b$, the AM becomes $b\frac {1+n^2}2$ and the HM becomes $\frac {2n^2b}{1+n^2}$ To make these integral we need $2(n^2+1)|b$ if $n$ is odd or $(n^2+1)|b$ if $n$ is odd. To make the numbers small, we take $b$ as small as possible. The minimal $|a-b|$ is then $2\cdot 2^2+2=10$ or $2 \cdot (3^2-1)=80$ as we found above
In order for the arithmetic mean to be integer, the difference between $a$ and $b$ must be even, and in order for the geometric mean to be integer, the difference between $a$ and $b$ must be some multiple of a difference of squares.
This means that for a working pair $a<b$, the difference can't be less than $4\sqrt a$, because the difference between adjacent same-parity squares will be at least this much, and working at multiples (>2) of the difference of squares will increase it further.
In order to prove the proposed minimum difference of $30$ in Ross' answer, therefore, it's enough to check pairs with the smaller value up to $56$, which I have done.
Of course, the limit there is hindsight. I actually checked numbers up to 40000 and reported back the 48 cases where the resulting calculations were co-prime. In each case this "base pair" was created by taking two small squares, multiplying both by two if necessary to get the difference even, then multiplying both by the average of the two. $(5,45)$ is generated by $(1,9)$; $(10,40)$ comes from $(1,4)$. The original pair found in the question is a doubled version of $(5,45)$, of course.
Put $$m:={a+b\over2}, \quad h:={2ab\over a+b}\ .$$ Write $m=r^2m'$, where $m'$ is square-free. Since we want $ab=hm=hr^2 m'$ to be a square it follows that $h=s^2 m'$.
Now $a$ and $b$ are the solutions of the quadratic equation $$x^2-2mx +hm=0\ ,$$ whence $$a=m-\sqrt{m^2-hm},\quad b=m+\sqrt{m^2-hm}\ .$$ It follows that $$m^2-hm=r^4m'^2-s^2m'\cdot r^2m'=r^2 m'^2(r^2-s^2)$$ should be a square as small as possible, which immediately implies $m'=1$. It remains to minimize the quantity $q:=r\sqrt{r^2-s^2}$ with nontrivial integer $\sqrt{r^2-s^2}$. This $q$ can be interpreted as product of a leg of a pythagorean triangle with the hypotenuse. I venture that the smallest such product is $3\cdot 5=15$. This leads to $a=10$, $\>b=40$, so that the minimal possible difference is $b-a=30$.
