The matrix $A=\begin{bmatrix}\cos(\theta)&\sin(\theta)\\-\sin(\theta)&\cos(\theta)\end{bmatrix}$ is a particularly important example and represents a clockwise rotation transformation by an angle of $\theta$.
You have correctly calculated the result, identified, or remembered the property that having repeatedly applied a rotation of $\theta$ some $n$ times successively that this is the same result as having gone through a single rotation of an angle of $n\theta$. That is to say $A^n = \begin{bmatrix}\cos(n\theta)&\sin(n\theta)\\-\sin(n\theta)&\cos(n\theta)\end{bmatrix}$.
Next, you correctly noted that $\dfrac{A^n}{n}=\begin{bmatrix}\frac{\cos(n\theta)}{n}&\frac{\sin(n\theta)}{n}\\ -\frac{\sin(n\theta)}{n}&\frac{\cos(n\theta)}{n}\end{bmatrix}$
From here, to compute $\lim\limits_{n\to\infty}\dfrac{A^n}{n}$, it suffices to compute the limits of each individual entry.
Note that regardless the value of $n$ and regardless the value of $\theta$ we have $-1\leq \cos(n\theta)\leq 1$. Similarly for $\sin(n\theta)$ and $-\sin(n\theta)$.
As a result since for all values of $n$ and all values of $\theta$ we have $\dfrac{-1}{n}\leq \dfrac{\cos(n\theta)}{n}\leq \dfrac{1}{n}$ and both of the outer limits both approach zero as $n$ grows large, so too must the limit of the center expresion approach zero as $n$ grows large as per the squeeze theorem. The proofs for the other entries are identical.
It follows then that $\lim\limits_{n\to\infty}\dfrac{A^n}{n}=\begin{bmatrix}0&0\\0&0\end{bmatrix}$, as you expected.
\left[and\right[instead of[and]makes the size of those delimiters scale so that it matches with the size of whatever they are delimiting. I warmly recommend that you learn to use them. Makes the output a lot nicer. I also changed\fracto\dfrac. That stands for Displayed FRACtion. You see that the are quite a bit larger. Basically both the numerator and the denominator are then in normal font size as opposed to getting shrunk. This is to spare the eyes of anyone over 50 years of age :-). – Jyrki Lahtonen Apr 14 '18 at 05:09\fracto\dfracis a style preference of mine. If you disprove, just tell me, and switch it back. I won't impose my style preference over anyone else. – Jyrki Lahtonen Apr 14 '18 at 05:10