\begin{align*}
A = \begin{vmatrix}
1+a^2-b^2 & 2ab & -2b \\
2ab & 1-a^2+b^2 & 2a \\
2b & -2a & 1-a^2-b^2
\end{vmatrix} &= (1+a^2+b^2)^3
\end{align*}
To get $1+a^2+b^2$ in the first row, we can apply $R_1 \rightarrow R_1+bR_3$. This will give
\begin{align*}
A = \begin{vmatrix}
1+a^2+b^2 & 0 & -b(1+a^2+b^2) \\
2ab & 1-a^2+b^2 & 2a \\
2b & -2a & 1-a^2-b^2
\end{vmatrix}
\end{align*}
Taking $1+a^2+b^2$ out from the first row, we get
\begin{align*}
A = (1+a^2+b^2)\begin{vmatrix}
1 & 0 & -b \\
2ab & 1-a^2+b^2 & 2a \\
2b & -2a & 1-a^2-b^2
\end{vmatrix}
\end{align*}
We need one more zero in the first row. This can be obtained by taking $C_3 \rightarrow C_3+bC_1$. This gives
\begin{align*}
A = (1+a^2+b^2)\begin{vmatrix}
1 & 0 & 0 \\
2ab & 1-a^2+b^2 & 2a(1+b^2) \\
2b & -2a & 1-a^2+b^2
\end{vmatrix}
\end{align*}
Now, expanding by the first row, we obtain
\begin{align*}
A &= (1+a^2+b^2)[(1-a^2+b^2)^2 + 4a^2(1+b^2)] \\
&= (1+a^2+b^2)[(1+b^2-a^2)^2 + 4a^2(1+b^2)] \\
&= (1+a^2+b^2)[(1+b^2)^2 + a^4 -2a^2(1+b^2)+ 4a^2(1+b^2)] \\
&\qquad \qquad \qquad \qquad \text{expanding } ((1+b^2) - a^2)^2 \\
&= (1+a^2+b^2[(1+b^2)^2 + a^4 +2a^2(1+b^2)] \\
&= (1+a^2+b^2)(1+a^2+b^2)^2 \\
&= (1+a^2+b^2)^3
\end{align*}
Since $a^2-b^2 = 2$, the determinant can be written as
$$ (1+ b^2 + 2 + b^2)^3 = (3+2b^2)^3 $$
This is least when $b^2$ is least. The least value of $b^2$ is 0 and hence the least value is 27.