Find the matrix of the rotation in $\Bbb R^3$ through the angle $\alpha$ around the vector $(1, 2, 3)^T$ . We assume that the rotation is counterclockwise if we sit at the tip of the vector and looking at the origin.
Let $P(\vec x)$ be the linear transformation that rotates the coordinate system such that $z$ axis is in the direction of $(1, 2, 3)^T$, then by using some trignometry,
$P((0,0, 1)^T) = \dfrac1{\sqrt{13}}(1,2,3)^T$, $P((0,1, 0)^T) =(0,\sec \theta,\tan \theta)^T$ and $P((1,0,0)^T) =(\sec \theta,0,\tan \theta)^T$, where $\theta$ is the angle between $(1,2,3)^T$ and $(0,0,1)^T$.
Solving for $\theta$ and using matrix representation of $P$,
$$P(\vec x) = \begin{bmatrix}\dfrac{\sqrt{13}}{3} & 0 &\dfrac1{\sqrt{13}} \\ 0& \dfrac{\sqrt{13}}{3}& \dfrac2{\sqrt{13}} \\\dfrac{-2}{3} & \dfrac{-2}{3}& \dfrac3{\sqrt{13}} \end{bmatrix}\vec x$$
Therefore, I got the final rotation transformation as $T(\vec x) := (P^{-1}\circ R_\alpha \circ P)(\vec x)$, where $$R_\alpha(\vec x) = \begin{bmatrix}\cos \alpha & -\sin \alpha & 0 \\ \sin \alpha & \cos \alpha & 0 \\ 0&0&1\end{bmatrix}.$$
Basically what I did was to rotate the coordinate system with $P$ such that $z$-axis and $(1,2,3)^T$ coincide, then I rotated $xy$-plane by $\alpha$ and then reverted back to original coordinates.
Is this correct ?
Please don't use quaternions or any other voodoo like given here. Please use basic trignometry and linear algebra.
Sorry, normally I don't ask this type of question but I can't verify my results in this case.