Evaluate $b \log(a)$ and $a \log(b)$. Suppose, for example, that $a \log(b)$ is smaller. $b \log(a) - a \log(b)=\log(\frac{a^b}{b^a})$. If this is so large that exponentiating causes overflow (or even gives you a 20-digit number), then for all practical purposes $x=a^b-b^a$ is just $a^b$ for purposes of the number of significant digits you have, and you are done (i.e., $\log(x)=b \log(a)$ to avoid overflow). If $b \log(a) - a \log(b)$ is less than $\log(10^{20})$ or so, exponentiate it and subtract $1$ to get $\frac{x}{b^a}$. If $b^a$ is not too large, solve directly. If $b^a$ is very large, take the log again to get $\log(x)-a\log(b)$, and then add $a\log(b)$ to get $\log(x)$.
The key thing is to realize that is $b \log(a)$ is sufficiently larger than $a \log(b)$, then $a^b-b^a$ will just be $a^b$. And of course (which I think you already know) to realize that if $x$ would overflow, you would still like to return $\log (x)$.