Solution: The minimum value is $125$ at $a = 5$ and $b = 10$.
Approach: First let's express $b$ in terms of $a$. From $\frac{1}{a} + \frac{8}{b} = 1$, we have
$$1 - \frac{1}{a} = \frac{8}{b} \implies \frac{a - 1}{8a} = \frac{1}{b}$$
$$\implies b = \frac{8a}{a-1}$$
and now since $b$ must be $> 0$, we have $\frac{8a}{a - 1} > 0 \implies a > 1$ which is stronger than $a > 0$ (from problem statement).
Thus, all that is left to find is the minimum of the equation $a^2 + b^2$ or, $f(a) = a^2 + \frac{64 a^2}{(a - 1)^2}$ where $a>1$. This minimum (which happens to be 125) can be easily found via differentiation.
Differentiation: The required minima will be obtained when $\frac{df(a)}{da} = 0$ where $a > 0$.
$$\frac{df(a)}{da} = 0,\ \text{where } a > 0$$
$$\implies \frac{2a(a-5)(a(a + 2) + 13)}{(a-1)^3} = 0,\ \text{where } a > 0$$
$$\implies (a - 5) > 0 \implies a = 5$$
$$\implies a = 5,\ b= \frac{8\times 5}{4} = 10$$
Thus, the minimum is obtained at $a = 5, b = 10$ and the value is $a^2 + b^2 = 25 + 100 = 125$ as needed.
Note that I have skipped some steps (when coming to $a - 5 > 0$) and arguments (prove that the point $a = 5$ is indeed a minima for $f(a)$ using $f''(a)$ at $a = 5$) here.