Lets say you have two column vectors: $$\begin{array}{|r r|} 3 & -4 \\ 4 & 3 \\ \end{array}$$
How to find the scaling of this matrix? I thought that it can be any number but apparently it is $5$ which helps to find the rotational angle for this matrix with $$\begin{array}{|r r|} \cos & -\sin \\ \sin & \cos \\ \end{array}$$
For this example the angle would be $\arccos(0.6)$ counterclockwise? Why is the scaling $5$? Why not $4$ or $3$ or $2$?
One way to see this would be to see how the matrix affects the vectors $[1\; 0]$ and $[0\; 1]$. For instance, if we call your matrix $A$, then we see that $$ A \begin{pmatrix} 1\\0\end{pmatrix} = \begin{pmatrix} 3\\4\end{pmatrix}$$ and $$ A \begin{pmatrix} 0\\1\end{pmatrix} = \begin{pmatrix} -4\\3\end{pmatrix}.$$ How did these lengths change? We took $[1\; 0]$, a vector of length $1$, and $A$ output a vector of length $5$. Similarly, $[0\; 1]$ yielded another vector of length $5$.
Since every vector in $\mathbb{R}^2$ is a unique sum of $[1\; 0]$ and $[0\; 1]$, each of which is scaled by a factor of $5$, and as $A$ is linear, we must have that $A$ scales every vector by length $5$. Thus $A$ is of the form $$ A = \begin{pmatrix} 5&0\\0&5\end{pmatrix} O$$ where $O$ is some orthogonal matrix (a rotation/reflection matrix in this case). $\diamondsuit$
I didn't use hint you were given because it's unnecessary. But you could in principle first find the rotation matrix (i.e. the matrix I called $O$ above), and then invert it to find the scaling factor.
We scale the vectors by $5$ because we want them to have length $1$. If the coordinates of the vector are $a$ and $b$, we wish to scale the vector by $c>0$ such that $\sqrt{(ca)^2+(cb)^2}=1$ (this is the length of the vector). In the first case we have $a=3$ and $b=4$ for the first vector, so the equation becomes $\sqrt{16c^2+9c^2}=1\implies 5c=1\implies c=\dfrac{1}{5}$. The second case is the same.
