Rewrite the generic term to average as
$$|\cos(n^2)| = \left|\cos\left(\pi \frac{n^2}{\pi}\right)\right| =
\left|\cos\left(\pi \left\{\frac{n^2}{\pi}\right\}\right)\right|$$
where $u_n = \frac{n^2}{\pi}$ and $\{\cdot\}$ stands for the fractional part.
We know $\pi$ is irrational. So for any positive integer $h$,
$\frac{2h}{\pi}$ is irrational.
By equidistribution theorem, the sequence
$$\{ u_{n+h} - u_n \} = \left\{\frac{2h}{\pi} n + \frac{h^2}{\pi}\right\}$$
is equidistributed modulo $1$.
Since this is true for all positive integer $h$, van der Corput's difference theorem tells us $u_n$ is also equidistributed modulo $1$.
Recall for any Riemann integrable function $f : [a,b] \to \mathbb{R}$ and any sequence $(s_1,s_2,\ldots)$ equidistributed on $[a,b]$, we have
$$\lim_{N\to\infty} \sum_{n=1}^N f(s_n) = \frac{1}{b-a}\int_a^b f(x)dx$$
Apply this to $f(x) = |\cos(\pi x)|$ and $s_n = u_n$, we obtain
$$\lim_{N\to\infty} \frac1N \sum_{n=1}^N|\cos(n^2)| = \int_0^1 |\cos(\pi x)| dx = \frac{2}{\pi}$$