Question
If P is the plane through $(1/2, -1, 0)$ orthogonal to $(0, 1, 0)$ find an isometry F = TC such that F(P) is the plane through $(1, -2, 1)$ orthogonal to $(1, 0, -1)$.
This is an exercise in O'neill's Elementary Differential Geometry.
I found the given plane P: y = -1 and F(P): x-z=$0$
But that's all I did. How should I approach this problem?
First of all, I agree with your plane equations. The first one is affine whereas the second one is linear (RHS=0).
Let us reduce this affine issue into a linear one by bringing affine plane $y=-1$ onto vector plane $y=0$ (using a translation $T$ by vector $V_T=(0,1,0)^T$) according to the following classical pattern :
$$F(X)=S(X+V_T)-V_T\tag{1}$$
to be understood as the composition :
$$\begin{matrix}X&\overset{F}{\longrightarrow}& S(X+V_T)-V_T & \text{affine level}\\\downarrow && \uparrow\\X+V_T &\longrightarrow & S(X+V_T) & \text{linear level} &\end{matrix}.$$
Therefore, let us work on the associated linear subspaces $P_1$ and $P_2$ with resp. normal vectors $N_1=(0,1,0)^T$ and $N_2=(1,0,1)^T/\sqrt{2}$ (please note the second vector has been normalized).
Instead of a rotation, we can look for a symmetry. This symmetry is with respect to one of the two bissector planes $P_3$ or $P'_3$ of $P_1$ and $P_2$, with resp. normals $N_1+N_2$ and $N_1-N_2$.
Let us consider plane $P_3$ with normal $N_1+N_2 =(1/\sqrt{2} , 1, 1/\sqrt{2})^T$ whose unit norm version is $N_3=(1/2,1/\sqrt{2},1/2)^T$.
Now, use the Householder formula of the symmetry matrix with respect to $P_3$:
$$S=I_3-2N_3N_3^T=\begin{pmatrix}1&0&0\\0&1&0\\0&0&1\end{pmatrix}-2\begin{pmatrix} 1/4&a&1/4\\a&1/2&a\\1/4&a&1/4\end{pmatrix}$$ where
$$a:=1/(2\sqrt{2}).$$
Finally :
$$S=\begin{pmatrix}1/2&-1/\sqrt{2}&-1/2\\-1/\sqrt{2}&0&-1/\sqrt{2}\\-1/2&-1/\sqrt{2}&1/2\end{pmatrix}.$$
It remains to apply (1).
Remark : If $N_1-N_2$ had been taken instead of $N_1+N_2$, we would have obtained different symmetry.
