For a transformation of variables, the curve equations suggest something reasonably straightforward:
$$ u \ = \ x^2 + y^2 \ \ \rightarrow \ \ u = 8 \ , \ u = 16 \ \ ; \ \ v \ = \ x^2 - y^2 \ \ \rightarrow \ \ v = 2 \ , \ v = 8 \ \ ; $$
$$ A(2 \sqrt{2} \ , \ 0 ) \ \rightarrow \ (8 \ , \ 8) \ \ , \ \ B(2 \sqrt{3} \ , \ 2 ) \ \rightarrow \ (16 \ , \ 8) \ \ , $$
$$ C(3 \ , \ \sqrt{7} ) \ \rightarrow \ (16 \ , \ 2) \ \ , \ \ D(\sqrt{5} \ , \ \sqrt{3} ) \ \rightarrow \ (8 \ , \ 2) \ \ . $$
Our integration is now a rectangle in the $ \ uv-$ plane with the "circulation" from $ \ A \ $ around to $ \ D \ $ and back to $ \ A \ $ in the "clockwise", rather than the "counter-clockwise" direction.

The double integral is transformed as
$$ \iint_{\mathbf{R}} \ (x^2 + y^2) \ dx \ dy \ \ \rightarrow \ \ \iint_{\mathbf{R^{\star}}} \ u \ · \ | \mathfrak{J} | \ \ du \ dv \ \ , $$
with $ \ \mathfrak{J} \ $ being the Jacobian matrix. It will be easier to compute the inverse Jacobian and find the determinant from that:
$$ \mathfrak{J}^{-1} \ = \ \left| \begin{array}{cc}
\frac{\partial u}{\partial x} & \frac{\partial u}{\partial y} \\
\frac{\partial v}{\partial x} & \frac{\partial v}{\partial y}
\end{array} \right| \ \ = \ \ \left| \begin{array}{cc}
2x & 2y \\
2x & -2y
\end{array} \right| \ \ = \ \ -8xy \ \ \Rightarrow \ \ \mathfrak{J} \ = \ -\frac{1}{8xy} \ \ , $$
the minus-sign indicating that the orientation of the region in the $ \ uv-$ plane is reversed from that in the $ \ xy-$ plane, as we remarked earlier. We can solve the set of variable transformation equations to obtain
$$ u \ = \ x^2 + y^2 \ \ , \ \ v \ = \ x^2 - y^2 \ \ \Rightarrow \ \ 2x^2 \ = \ u + v \ \ , \ \ 2y^2 \ = \ u - v $$
$$ \Rightarrow \ \ 4 x^2 y^2 \ = \ u^2 - v^2 \ \ \Rightarrow \ \ 2xy \ = \ \sqrt{u^2 - v^2} \ \ . $$
Our transformed integral is thus
$$ \int_2^8 \int_8^{16} \ u \ · \ \left| \ -\frac14 · \frac{1}{\sqrt{u^2 - v^2}} \ \right| \ \ du \ dv \ \ = \ \ \frac14 \ \int_2^8 \int_8^{16} \ \frac{u}{\sqrt{u^2 - v^2}} \ \ du \ dv $$
$$ = \ \ \frac14 \ \int_2^8 \ \sqrt{u^2 - v^2} \left \vert_8^{16} \right. \ \ dv \ \ = \ \ \frac14 \ \int_2^8 \left( \ \sqrt{16^2 - v^2} \ - \ \sqrt{8^2 - v^2} \ \right) \ \ dv $$
$$ = \ \ \frac14 · \frac12 \ \left[ \ v \sqrt{16^2 - v^2} \ + \ 16^2 · \arcsin \left( \frac{v}{16} \right) \ - \ v \sqrt{8^2 - v^2} \ - \ 8^2 · \arcsin \left( \frac{v}{8} \right) \ \right] \left \vert_2^8 \right. $$
$$ = \ \ 8 \sqrt{3} \ + \ \frac{16}{3} \pi \ - \ 0 \ - \ 4 \pi \ - \ \frac32 \sqrt{7} \ - \ 32 · \arcsin \left( \frac18 \right) \ + \ \frac12 \sqrt{15} \ + \ 8 · \arcsin \left( \frac14 \right) $$
$$ = \ \ 8 \sqrt{3} \ + \ \frac{4}{3} \pi \ - \ \frac32 \sqrt{7} \ - \ 32 \arcsin \left( \frac18 \right) \ + \ \frac12 \sqrt{15} \ + \ 8 \arcsin \left( \frac14 \right) $$
$$ \approx \ \ 13.8564 \ + \ 4.1888 \ - \ 3.9686 \ - \ 4.0105 \ + \ 1.9365 \ + \ 2.0214 \ \ \approx \ \ \mathbf{14.0240} \ \ . $$
The two integral calculations in Cartesian and polar coordinates confirm this result.
$$ \ \ $$
$ \mathbf{APPENDIX -} $ We would like to verify that all three exact expressions are in fact equivalent. From the three calculations, we obtained
$ \mathbf{Cartesian :} \quad -\frac32 \sqrt{7} \ + \ \frac12 \sqrt{15} \ + \ 8 \sqrt{3} \ + \ \frac{40}{3} \pi \ - \ 64 \arcsin \left( \frac34 \right) \ + \ 16 \arcsin \left( \frac{\sqrt{5}}{2 \sqrt{2}} \right) $
$ \mathbf{polar :} \quad 8 \sqrt{3} \ - \ \frac{32}{3} \pi \ - \ 16 \arctan \left( \sqrt{\frac35} \right) + \ 64 \arctan \left( \frac{\sqrt{7}}{3} \right) + \ \frac12 \sqrt{15} \ - \frac32 \sqrt{7} $
$ \mathbf{uv-plane :} \quad 8 \sqrt{3} \ + \ \frac{4}{3} \pi \ - \ \frac32 \sqrt{7} \ - \ 32 \arcsin \left( \frac18 \right) \ + \ \frac12 \sqrt{15} \ + \ 8 \arcsin \left( \frac14 \right) $
The three "square-root" terms appear in all of these, so we only need to examine the trigonometric terms.
In comparing the Cartesian and polar terms, we note that
$$ \alpha \ = \ \arcsin \left( \frac34 \right) \ = \ \arctan \left( \frac{3}{\sqrt{7}} \right) \ \ \text{and} \ \ \beta \ = \ \arcsin \left( \sqrt{\frac58} \right) \ = \ \arctan \left( \sqrt{\frac53} \right) \ \ . $$
So $ \ \frac{\pi}{2} - \alpha \ = \ \arctan \left( \frac{\sqrt{7}}{3} \right) \ $ and $ \ \frac{\pi}{2} - \beta \ = \ \arctan \left( \sqrt{\frac35} \right) \ \ . $ The three terms of interest in the polar-integral expression are then
$$ - \ \frac{32}{3} \pi \ - \ 16 · \left[ \frac{\pi}{2} - \beta \right] + \ 64 · \left[ \frac{\pi}{2} - \alpha \right] \ = \ \left( \frac{-32-24+96}{3} \right) \pi \ + \ 16 · \beta \ - \ 64 · \alpha $$
$$ = \ \frac{40}{3} \pi \ - \ 64 · \alpha \ + \ 16 · \beta \ \ ; $$
hence, the Cartesian and polar expressions are equivalent.
Checking the $ \ uv-$ plane result is somewhat tricky, in that it isn't immediately obvious what terms in the expressions should be "matched up". It turns out that for the Cartesian and $ \ uv-$ plane expressions that a "double-angle" with the same "added angle" works out:
$$ \frac{40}{3} \pi \ - \ 64 · \alpha \ + \ 16 · \beta \ \ = \ \ \frac{4}{3} \pi \ - \ 32 · ( 2 \alpha + \phi) \ + \ 8 · ( 2 \beta + \phi) $$
$$ \Rightarrow \ \ \frac{36}{3} \pi \ \ = \ -32 \phi \ + \ 8 \phi \ \ \Rightarrow \ \ 12 \pi \ = \ -24 \phi \ \ \Rightarrow \ \ \phi \ = \ -\frac{\pi}{2} \ \ . $$
Since we wish to compare sine values, we will need
$$ \ \sin \left(2 \theta - \frac{\pi}{2} \right) \ = \ \sin (2 \theta) · \cos \left( \frac{\pi}{2} \right) \ - \ \cos (2 \theta) · \sin \left( \frac{\pi}{2} \right) \ = \ - \cos (2 \theta) \ = \ \sin^2 \theta \ - \ \cos^2 \theta \ \ . $$
From the statements above, we have
$$ \sin \alpha \ = \ \frac34 \ , \ \cos \alpha \ = \ \frac{\sqrt{7}}{4} \ , \ \sin \beta \ = \ \sqrt{\frac58} \ , \ \sin \beta \ = \ \sqrt{\frac38} \ \ , $$
hence,
$$ \ \sin \left(2 \alpha - \frac{\pi}{2} \right) \ = \ \left( \frac34 \right)^2 \ - \ \left(\frac{\sqrt{7}}{4} \right)^2 \ = \ \frac{9 - 7}{16} \ \ = \ \ \frac18 \ \ \text{and} $$
$$ \ \sin \left(2 \beta - \frac{\pi}{2} \right) \ = \ \left( \sqrt{\frac58} \right)^2 \ - \ \left(\sqrt{\frac38} \right)^2 \ = \ \frac{5 - 3}{8} \ \ = \ \ \frac14 \ \ . $$
With the correspondences indicated between the angles in each expression, we conclude that the exact expressions found for the double integral by each method are indeed equivalent.