We can use the inequalities between quadratic mean (a.k.a. square root mean), arithmetic mean, geometric mean and harmonic mean
$$\sqrt{\frac{x^2+y^2}2} \ge \frac{x+y}2 \ge \sqrt{xy} \ge \frac2{\frac1x+\frac1y}.$$
These inequalities are true for any $x,y>0$. The equality holds if and only if $x=y$. They can be generalized for more than two variables (also proofs for two variables are easier). If you are not familiar with these inequalities, you can find a lot of material on them, just try to search for some reasonable queries like this one or this one.
You want to minimize $f(a,b)=\left(a+\frac1a\right)^2+\left(b+\frac1b\right)^2$. Notice that
$$f(a,b)=\left(a+\frac1a\right)^2+\left(b+\frac1b\right)^2 = a^2+b^2 + \frac1{a^2}+\frac1{b^2} +4.$$
So minimizing $g(a,b)=a^2+b^2 + \frac1{a^2}+\frac1{b^2}$ is an equivalent problem.
Using QM-AM inequality we get
$$\frac{a^2+b^2}2 \ge \left(\frac{a+b}2\right)^2 = \frac14$$
which implies $$a^2+b^2 \ge \frac12.\tag{1}$$
If we combine HM-GM and AM-GM we get
$$\frac2{\frac1{a^2}+\frac1{b^2}} \le \sqrt{a^2b^2} = ab \le \left(\frac{a+b}2\right)^2 = \frac14,$$
which is equivalent to
$$\frac1{a^2}+\frac1{b^2} \ge 8.\tag{2}$$
By adding the inequalities $(1)$ and $(2)$ we get
$$g(a,b) = a^2+b^2 + \frac1{a^2}+\frac1{b^2} \ge 8+\frac12$$
and
$$f(a,b)=g(a,b)+4 \ge 12+\frac12 = \frac{25}2.$$
The inequalities $(1)$ and $(2)$ can be also interpreted geometrically.
Minimizing $a^2+b^2$ for $a+b=1$ is simply finding the point on the line $a+b=1$ which is closest to the origin. If you draw the picture, you immediately see that it is the point given by $a=b=\frac12$.
We also want some geometrical insight into minimizing $\frac1{a^2}+\frac1{b^2}$ for $a+b=1$. Let us transform this problem a bit.
If we denote $x=\frac1a$, then $y=\frac1b=\frac1{1-a}=\frac1{1-\frac1x}=\frac{x}{x-1}$. It is not difficult to see that this is a hyperbole with the asymptotes $x=1$ and $y=1$. And we want to minimize $\frac1{a^2}+\frac1{b^2}=x^2+y^2$, which means that we want to find the closest point to the origin on this hyperbole. (To be more precise, only in the part of the hyperbole in the first quadrant, since $x,y>0$.) Here is a plot from WolframAlpha.

Again we see from the picture that the closest point will be the one with $x=y$. (Which gives us $x=y=2$ and $a=b=\frac12$.)