ADDED: conclusion without the matrices:
When $k^2 - 4 k - 8 = 0,$ we get factoring over the reals because
$$ \color{magenta}{ (x+2y+z)^2 - \frac{1}{2} \left( 2y + (2-k) z \right)^2 } $$
is your quadratic form, and then we can factor as
$$ T^2 - \frac{1}{2} U^2 = \left(T + \frac{U}{ \sqrt 2}\right) \left(T - \frac{U}{ \sqrt 2}\right) $$
when $T= x+2y+z$ and $U= 2y + (2-k) z.$ Note that $2-k$ simplifies when $k$ is one of the two roots of $k^2 - 4 k - 8 = (k-2)^2 - 12.$ Thus we have either $k = 2 + 2 \sqrt 3$ or $k = 2 - 2 \sqrt 3$
ORIGINAL::
we write a quadratic form as $X^T H X,$ where capital $X$ is the column vector with elements $x,y,z$ and $X^T = (x,y,z).$ Here, $H$ is the Hessian matrix or half of that, for convenience I'm taking half this time.
The form factors only if the determinant of $H$ is zero.
$$
H =
\left(
\begin{array}{ccc}
1&2&1 \\
2&2&k \\
1&k&-5 \\
\end{array}
\right)
$$
and
$$ \det H = 8 + 4 k - k^2 $$
This becomes zero when
$$ k = 2 \pm 2 \sqrt 3 $$
To make this concrete, I will display $Q^T D Q = H,$ where $D$ is diagonal and $\det Q = 1.$
$$
\left(
\begin{array}{ccc}
1&0&0 \\
2&1&0 \\
1&\frac{2-k}{2}&1 \\
\end{array}
\right)
\left(
\begin{array}{ccc}
1&0&0 \\
0&-2&0 \\
0&0&\frac{k^2 - 4k - 8}{2} \\
\end{array}
\right)
\left(
\begin{array}{ccc}
1&2&1 \\
0&1&\frac{2-k}{2} \\
0&0&1 \\
\end{array}
\right)=
\left(
\begin{array}{ccc}
1&2&1 \\
2&2&k \\
1&k&-5 \\
\end{array}
\right)
$$
When $k^2 - 4k - 8$ is nonzero, this expresses your quadratic form as a sum of three squares of linear terms, with coefficients.
When $k^2 - 4 k - 8 = 0,$ we get factoring over the reals because
$$ (x+2y+z)^2 - 2 \left( y + \frac{2-k}{2} z \right)^2 $$
is your quadratic form, and then we can factor
$$ V^2 - 2 W^2 = (V + W \sqrt 2)(V - W \sqrt 2) $$
$\begin{align} b + q =4\ bq = 2\ c + r = 2\ cq + br = 2k\ cr =-5 \end{align}$
The short way is to use a CAS like wolframalpha.
– Jan-Magnus Økland Nov 01 '19 at 14:52