Find the unit vector $\vec{b}$ if $-\hat{i}+\hat{j}-\hat{k}$ bisects the angle between $\vec{b}$ and $\vec a=3\hat{i}+4\hat{j}$
The unit vector along the angle bisector is
$$\frac{-\hat{i}+\hat{j}-\hat{k}}{\sqrt3}=\frac{\vec{b}+\left(\frac35\hat{i}+\frac45\hat{j}\right)}{|\vec{b}+\frac35\hat{i}+\frac45\hat{j}|}$$
Im not able to extract $\vec b$ from here
You are correct that if $ \ \vec b \ $ is a unit vector and $- \hat i + \hat j - \hat k$ is angle bisector of $\vec b$ and $\vec a \ (= 3 \hat i + 4 \hat j)$, we have
$ \displaystyle \frac{-\hat{i}+\hat{j}-\hat{k}}{\sqrt3}=\frac{\vec{b}+\left(\frac35\hat{i}+\frac45\hat{j}\right)}{|\vec{b}+\frac35\hat{i}+\frac45\hat{j}|}$
Let's say $ \displaystyle |\vec{b}+\frac35\hat{i}+\frac45\hat{j}| = m$ where $m$ is magnitude.
Then, $ \displaystyle \vec b = \ \left(-\dfrac{m}{\sqrt3} - \dfrac{3}{5}\right)\hat i + \left(\dfrac{m}{\sqrt3} - \dfrac{4}{5}\right)\hat j -\dfrac{m}{\sqrt3} \hat k \ $ ....$(i)$
and $ \left(-\dfrac{m}{\sqrt3} - \dfrac{3}{5}\right)^2 + \left(\dfrac{m}{\sqrt3} - \dfrac{4}{5}\right)^2 + \dfrac{m^2}{3} = 1$
Simplifying, $m^2 - \dfrac{2m}{5\sqrt3} = 0$
and we get $m = \dfrac{2}{5\sqrt3}$
Substituting $m$ in $(i)$, $ \ \vec b = - \dfrac{1}{15} (11 \hat i + 10 \hat j + 2 \hat k)$
Because a rhombus’s diagonals bisect its interior angles, in general, a bisector of the angle between vectors $\mathbf u$ and $\mathbf v$ is $$\hat{\mathbf u}+\hat{\mathbf v}.$$
Therefore, for our problem: there exists some positive $k$ for which $$\mathbf b+\frac1{\sqrt{3^2+4^2}}\begin{pmatrix} 3 \cr 4 \cr 0 \end{pmatrix}=k\begin{pmatrix} -1 \cr 1 \cr -1 \end{pmatrix}.$$ Since $\mathbf b$ is a unit vector, it must be that$$\left(-k-\frac35\right)^2+\left(k-\frac45\right)^2+\left(-k\right)^2=1\\k=-\frac2{15}.$$ ($k$ being negative means that, contrary to the problem statement, the given vector is actually an exterior bisector of the angle between $\mathbf b$ and $\mathbf a.$)
Thus, $$\mathbf b=-\frac1{15}\begin{pmatrix} 11 \cr 10 \cr 2 \end{pmatrix}.$$
$k$ actually equals positive $\frac2{15}$ (the preceding lines and final answer remain correct), so the given vector is indeed the specified bisector.
