Let $\mathbf{x},\mathbf{y},\mathbf{w}$ be the following 3-vectors:
$$\mathbf{x}=\begin{pmatrix}x_{1}\\ x_{2}\\ x_{3}\end{pmatrix}\qquad\mathbf{y}=\begin{pmatrix}y_{1}\\ y_{2}\\ y_{3}\end{pmatrix}\qquad \mathbf{w}=\begin{pmatrix}0\\0\\1\end{pmatrix}$$
The vector $\mathbf{y}$ has the same magnitude as $\mathbf{x}$ and also lies in the $\mathbf{xw}$ plane, only its angle to $\mathbf{w}$ is a function of angle between $\mathbf{x}$ and $\mathbf{w}$.
Is there an easy way on how to express $\mathbf{y}$ in terms of $\mathbf{x},\mathbf{w}$ and their angle?
I found one way which is very clumsy and complicated:
Find rotation axis (or normal to $\mathbf{xw}$ plane) via cross product: $a=\frac{\mathbf{x}\times\mathbf{w}}{|\mathbf{x}\times\mathbf{w}|}$
Find angle between $\mathbf{x,w}$ using dot product: $\theta=\cos^{-1}\frac{\mathbf{x}\cdot\mathbf{w}}{|\mathbf{x}|\cdot|\mathbf{w}|}$
Express rotation as a matrix (using some axis-angle representation): $R(a,\theta)$
Express $\mathbf{y}$ as $\mathbf{y}=R(a,f(\theta))\mathbf{x}$
These steps lead to a single formula for $\mathbf{y}$, which is quite complicated. I need however such formula to be able to take derivatives of $y$ with respect to $\mathbf{x}_{1,2,3}$ or possible parameter of $f$ (the function may be something like $f(\kappa,\theta)$).