I want to compute the angle of rotation phi which would make the z component of the last vector (0.35) zero - basically putting all of the vectors in the xy plane. I'm familiar with rotation matrices but I'm not sure how to work backwards. I would appreciate any suggestions about where to start.
A = (0.00,-0.69,0.00)
B = (0.00,0.69,0.00)
C = (0.77, 1.19, 0.35)
I've tried computing the angle between the z-axis and vector C, and rotating by that which got me pretty close to zero but not exactly.
Since you want the two first vectors to stay the same, it would seem like a good idea to rotate about the $y$-axis. That means we can forget the $y$-component of the third vector (it won't be used), and concentrate on the $xz$-components.
We want to rotate $(0.77,0.35)$ about the origin so that the second component disappears. The angle needed for this rotation is $$ -\arctan\frac{0.35}{0.77}\approx-0.427 $$ (The negative sign is there because we don't want the angle of the vector, we want the angle to rotate the vector. These are opposites.)
So the rotation matrix for this is $$ \begin{bmatrix}\cos( -0.427)&-\sin (-0.427)\\ \sin(0.427) & \cos(-0.427)\end{bmatrix} $$ Going back up into 3D, we get $$ \begin{bmatrix}\cos (-0.427)&0&-\sin( -0.427)\\ 0&1&0\\\sin(0.427 )&0& \cos(-0.427)\end{bmatrix} $$ If you actually use this matrix, the $z$-component won't be exactly $0$, it will be about $-0.0003$. That's because $0.427$ isn't the exact angle to rotate. It's a matter of rounding errors. Use more decimals of the $\arctan$ value, and it should go better.
