$$T(n) = nT(\sqrt{n}) + n^2, T(2) = 1$$
The master theorem does not apply here, only recursion tree. Assume $n = 2^{2^k}$.
The solution for self-check is $T(n = 2^{2^k}) = k \cdot n^2 + \frac{n^2}{4}$.
Edited:
My approach:
$T(2^{2^k}) = 2^{2^k}T(2^{2^{k-1}}) + 2^{2^{k + 1}}$
$= 2^{2^k + 2^{k - 1}}T(2^{2^{k-2}}) + 2^{2^{k+1}} + 2^{2^{k+1}}$
...
$=2^{\sum_{i = 1}^{k} 2^i} + k \cdot 2^{2^{k+1}}$
$=2^{\sum_{i = 0}^{k} 2^i - 1} + k \cdot 2^{2^{k+1}}$
$=2^{2^{k+1} - 2} + k \cdot n^2$
$=\frac{n^2}{4} + k \cdot n^2$