As the title says, this is a problem for a mathematics contest, no calculators allowed.
Contestants should solve this problem within a few minutes or so, therefore brute-force is a no-go.
I've tried substituting the constants:
$$\sqrt{a^2+(ab)^2+b^2}$$
I have the idea of using $$(a+b)^2=a^2+b^2+2ab$$
But I still can't see a way to make the calculation simple, please help!
Edit: Some people are saying that "a few minutes" is too short. In a math contest, if you finish all the other problems fast, you may have extra time to solve this particular problem, so don't be afraid to share your thoughts, even if it may take 5 or 10 minutes.