For the case when $a,b>0,$ I used AM-GM Inequality as follows that:
$\frac{(a+b+\frac{1}{ab})}{3} \geq (ab\frac{1}{ab})^\frac{1}{3}$
This implies that $(a+b+\frac{1}{ab})\geq 3$. Hence, the minimum value of $(a+b+\frac{1}{ab})$ is 3
But the answer is $2+\sqrt{2}$ ... how is it ?
If you want to use AM-GM for this, you need to ensure that the equality condition can be met along with the constraint, by "balancing coefficients". Illustrated below:
$$a+b+\frac1{ab} = a+b +\frac{1}{2\sqrt2 ab}+\left(1-\frac1{2\sqrt2}\right)\frac1{ab}$$
Now by AM-GM, $$ a+b +\frac{1}{2\sqrt2 ab}\ge \frac{3}{\sqrt2}$$
and for the remaining term we have again $$\left(1-\frac1{2\sqrt2}\right)\frac1{ab} \ge \left(1-\frac1{2\sqrt2}\right)\frac{2}{a^2+b^2} = 2-\frac{1}{\sqrt2}$$
Combining these results, we have $a+b+\dfrac1{ab} \ge 2 + \sqrt2$, with equality iff $a=b=\frac1{\sqrt2}$.
Added: The key here is of course knowing how to split the LHS, which is by noting that if for equality we need $a=b=\dfrac1{k~ab}$ and for the constraint we need $a^2+b^2=1$, what could be the value of $k$. The rest is then easy applications of AM-GM.
If $a,b>0$, one can make the substitution $u=ab,$ where $0<u\leq \frac 1 2$ (by AM-GM). Note that $$1=a^2+b^2=(a+b)^2-2ab,$$ hence $$a+b=\sqrt{1+2u}.$$ It follows that one needs to minimize $$f(u):=a+b+\frac 1{ab}=\sqrt{1+2u}+\frac 1 u, \qquad 0<u\leq \frac 1 2.$$
By Calculus, it is easy to see that $f$ has a global mimimum at $u_0\in (1,2)$ with $u_0^4-2u_0-1=0$, but $f$ is decreasing on $(0,\frac 1 2].$ It follows that $f$ achieves absolute minimum over $(0,\frac 1 2]$ at $u=\frac 1 2=ab,$ i.e. when $a=b=\frac{\sqrt{2}}2$ (the case of equality by AM-GM: $1=a^2+b^2\geq 2ab).$ Hence $$f(1/2)=2+\sqrt{2}$$ is the minimum of $a+b+\frac 1 {ab}.$ QED
$2ab = (a+b)^2 - (a^2+b^2)$, hence we need to find the minimum value of $a + b + \frac{2}{(a+b)^2 - (a^2+b^2)}$ $ = a + b + \frac{2}{(a+b)^2 - 1}$.
Let us find the maximum value (k) of $a+b$ given $a^2+b^2=1$. Thinking graphically, when $a+b = k$ is tangent to the circle, its gradient is $-1$ and hence the gradient of the normal is $1$, or $a=b$. Hence $k = \frac{1}{\sqrt2} + \frac{1}{\sqrt2} = \sqrt{2}$, which gives $\sqrt{2} + \sqrt{2} + \frac{2}{2-1} = 2 + \sqrt{2}$ as the answer.
This relies on the fact that $f(x) = x + \frac{2}{x^2 - 1}$ is decreasing for $x \in (1, \sqrt{2}]$. Partial fractions gives $f(x) = x + \frac{1}{x-1} - \frac{1}{x+1}$ and taking the derivative shows that $f'(x) < 0$ in the given interval. There might be a way without calculus.
Let the minimum value of $a+b+\frac{1}{ab} = k$. Both curves must be tangent to each other, which means they must have the same gradient. Thus $a^2 + b^2 = 1 \implies \frac{db}{da} = -\frac{a}{b}$, and:
$$a+b+\frac{1}{ab} = k \implies 1 + \frac{db}{da} - \frac{1 + db/da}{(ab)^2} = 0$$
$$\implies (ab)^2 \left(1 - \frac{a}{b} \right) - 1 - -\frac{a}{b} = 0$$ $$\implies a^2b^2 - a^3b - 1 + \frac{a}{b} =0$$ $$\implies a^2b^3 - a^3b^2 - b + a = 0$$ $$\implies a^2b^2(b-a) - (b-a) =0$$ $$\implies (ab+1)(ab-1)(b-a) = 0$$
but only $b-a = 0$ intersects with $a^2 + b^2 = 1$. This gives $a = b = \frac{1}{\sqrt2}$, and hence $k = \frac{1}{\sqrt2} + \frac{1}{\sqrt2} + \frac{1}{1/2} = 2 + \sqrt{2}$.
Constrant: $$q=x^2 +y^2-1=0$$
Goal: $$ c= x+ y + \frac{1}{xy}$$
Now, by LaGrange multipliers , we can say that:
$$ \nabla q = \lambda \nabla c$$
Hence,
$$2x = 1 - \frac{1}{x^2y} \tag{1}$$
$$ 2y = 1- \frac1{y^2 x} \tag{2}$$
By multiplication of $x$ in eqtn (1) and $y$ in eqtn (2):
$$ 2x^2 = x - \frac{1}{xy} \tag{3}$$
$$ 2y^2 = y - \frac{1}{yx} \tag{4}$$
Substract the two equations:
$$ 2(x^2 - y^2) = (x-y)$$
Now, either $x=y$ or $x+y = \frac12$, substitute this into the equation $(x+y)^2 -2xy -1 = 0$ while considering the constraints
$$ 2y^2 =1 \to (x, y)=\{(\frac{1}{\sqrt{2} }, \frac{1}{\sqrt{2} }) \}\tag5$$
$$ (\frac12)^2 -2xy -1 = 0 \to \frac{1}{xy}=-\frac83 \tag6$$
We can check that it is indeed eqtn (5) where the function extremizes with $2+ \sqrt{2}$
