Answer:
Velocity of the first boat $\hat V_1= 10\hat i$
Velocity of the second boat $\hat V_2 = 16cos(\frac{\pi}{3})\hat i + 16sin(\frac{\pi}{3})\hat j$
After a time "t"
Displacement by boat 1 $\hat D_1 = 10t\hat i$
Displacement by boat 2 $\hat D_2 = 16tcos(\frac{\pi}{3})\hat i + 16tsin(\frac{\pi}{3})\hat j$
Using parallelogram law to find the Displacement between Boat 1 and Boat 2
$\hat D = \hat D_1 - \hat D_2$
$\hat D = (10-16cos(\frac{\pi}{3}))t\hat i - 16tsin(\frac{\pi}{3})\hat j$
Magnitude of the displacement (r) =$\sqrt{\left({\left(10t-16tcos(\frac{\pi}{3})\right)}^2 + {\left(16tsin(\frac{\pi}{3})\right)}^2\right)}$
If you simplify, then you get $r = \sqrt{\left(100t^2 + 256t^{2}{(cos(\frac{\pi}{3}))}^2 - 2.10.16t^{2}cos(\frac{\pi}{3}) + 256t^{2}{(-sin(\frac{\pi}{3}))}^2\right)}$
$ r = \sqrt{\left(100t^2 + 256t^{2} - 2.10.16t^{2}cos(\frac{\pi}{3}) \right)}$
This is where you are getting the forumula that you mentioned in the book.
$ r = \sqrt{\left(356t^2 -2.10.16.t^{2}\frac{1}{2}\right)}$
$ r = \sqrt{\left(356t^2 - 160t^{2}\right)}$
$ r = \sqrt{196t^2} = 14t$
After two hours into the jounrney, the distance would be $=14\times2 = 28$
The rate at which it is increasing would be
$dr/dt = 14$ miles per hour.
you get the "Rate at which the distance is increasing when boats are 2 hours into the journey" = $14$ miles.
This is the "Physics" way of solving the problem.
Thanks
Satish