Note that the inequality you wanted to prove is equivalent to $(a+b)^2\leq 2$, so once you've obtained that $(a+b)^2=1+2ab$, the next step should be trying to prove that $1+2ab \leq 2$. This can be proven using the AM-GM inequality, which says that $2ab \leq a^2+b^2$, where the equality is taken when $a=b$. Thus $1+2ab \leq 1+a^2+b^2 = 2$.
In your attempt, you've actually reversed the logic a little bit by first declaring that $\sqrt{1+2ab}$ is real, then trying to prove the condition for its real-ness. This doesn't make sense. You can prove that $2ab\geq -1$ by rearranging $(a+b)^2\geq 0$ and using the fact that $a^2+b^2 = 1$. The equality here is taken when $a = -b$, which leads to the values $b = \pm \frac{1}{\sqrt{2}}$ that you solved for. In any case, this doesn't help you prove the original inequality, because it provides a lower-bound, not an upper bound to $(a+b)^2$.