You can solve this problem using polar decomposition. Write $f = o \circ p$ where $o \colon X \rightarrow X$ is orthogonal and $p \colon X \rightarrow X$ is self-adjoint and positive semi-definite. To show that $f$ is orthogonal, it is enough to show that $p = \operatorname{id}$.
Choose an orthonormal basis of eigenvectors $v_1, \dots, v_n$ of $p$ with $p(v_i) = \lambda_i v_i$ where $\lambda_i \geq 0$ and choose two distinct indices $1 \leq i < j \leq n$. Since $k \leq n-1$, we have $k - 1 \leq n - 2$ and so we can choose distinct indices $l_1, \dots, l_{k-1}$ such that $l_1,\dots,l_{k-1},i,j$ are all distinct. Then
$$ G(v_i,v_{l_1},\dots,v_{l_{k-1}}) = 1 $$
while
$$ G(f(v_i),f(v_{l_1}),\dots,f(v_{l_{k-1}})) = G(p(v_i), p(v_{l_1}), \dots, p(v_{l_{k-1}})) \\
= G(\lambda_i v_i, \lambda_{l_1} v_{l_1}, \dots, \lambda_{l_{k-1}} v_{l_{k-1}}) = \lambda_i^2 \cdot \prod_{r = 1}^{k-1} \lambda_{l_r}^2 $$
so
$$ \lambda_i^2 \prod_{r=1}^{k-1} \lambda_{l_r}^2 = 1. $$
In particular, $\lambda_{l_r} > 0$ for all $r$ and since the eigenvalues are non-negative, we get
$$ \lambda_i \prod_{r=1}^{k-1} \lambda_{l_r} = 1. $$
The same holds with $i$ replaced by $j$ and so we get
$$ \lambda_i \prod_{r=1}^{k-1} \lambda_{l_r} = \lambda_j \prod_{r=1}^{k-1} \lambda_{l_r} \implies (\lambda_i - \lambda_j) \prod_{r=1}^{k-1} \lambda_{l_r} = 0 \implies \lambda_i = \lambda_j. $$
Thus, we deduce that $\lambda_1 = \dots = \lambda_n$ and $\lambda_i^k = 1$ which implies that $\lambda_i = 1$ for all $1 \leq i \leq n$ so $p = \operatorname{id}$.
