If we write this as $$ H(\omega) = {a(\omega) \over b(\omega)}$$ then you can see that $b(\omega) H(\omega) = a(\omega)$. In particular, $\vert a(\omega)\vert = \vert b(\omega)H(\omega) \vert = \vert b(\omega) \vert \cdot \vert H(\omega) \vert$. This is one way to show $$ \vert H(\omega) = {\vert a(\omega) \vert \over \vert b(\omega) \vert } $$ so you can find your answer by taking magnitudes of numerator and denominator separately.
Now since each $a$ and $b$ is a sum the calculation is not completely straightforward; I would multiply by conjugates. That is, since $z \overline{z} = \vert z \vert ^2$, you can try
$$ \vert a(\omega) \vert^2 = a(\omega)\overline{a(\omega)} = (1 - \sqrt{2}e^{-j\omega} + e^{-2j\omega})(1 - \sqrt{2}e^{j\omega} + e^{2j\omega}) $$
$$ = 4 - \sqrt{2}(e^{j\omega} + e^{-j\omega}) + (e^{2j\omega}+e^{-2j\omega}) - \sqrt{2}(e^{j\omega}+e^{-j\omega})$$
and use the fact that $\displaystyle \cos(\omega) = {e^{j\omega} + e^{-j\omega} \over 2}$to begin calculating $|a(\omega)|$. Same works for $\vert b(\omega) \vert$.