rewrite $a_{n+1}=\frac{1}{4}[a_n(a_n+1)]+\frac12$ prove by induction that $a_n+1\ge4$ and $a_{n+1}>a_n\ge3$.
(Do not forget to use that $a_1=3$ for otherwise the proof won't work.)
Once you do that, if the sequence was convergent then both $a_n$ and $a_{n+1}$ would converge to the same limit $L>3$. You would get
$L=\frac{1}{4}(L^2+L+2)$. Solve this quadratic equation for $L$ and figure that the roots are too small to be a limit of numbers greater than $3$.
Alternatively, don't bother with the quadratic equation about $L$, but prove by induction that $a_n+1\ge4$, $a_{n+1}>a_n\ge3$, and $a_{n+1}\ge a_n+\frac12$. Then $a_{n+1} - a_n\ge \frac12$ for all $n$, which in particular implies that the sequence $a_n$ is not Cauchy, and cannot possibly converge. It also shows directly that $a_n\to\infty$, that is, $a_n$ diverges to infinity (which you could have also concluded, indirectly, using that $a_n$ is increasing but has no limit, after you found the possible roots for $L$).