You can use the pair $(\cos 2\alpha,\sin 2\alpha)$ to code $\alpha=\arctan M$.
EDIT:
The mention of a custom distance function made me think of computing a custom distance straight from the coordinates of the endpoints, rather than dealing with slopes and intercepts. One possibility that crossed my mind is to calculate how far the endpoints are from the line that passes through the midpoints of the segments. You can do that easily enough with the cross product.
Let $(x_1,y_1)$ and $(x_2,y_2)$ be the endpoints of one segment, $(x_3,y_3)$ and $(x_4,y_4)$ those of another segment. Let $(u,v)$ be the vector from the midpoint of first segment to the midpoint of the other one, that is,
$$\begin{eqnarray*}
u&=&\frac{x_3+x_4}{2}-\frac{x_1+x_2}{2},\\
v&=&\frac{y_3+y_4}{2}-\frac{y_1+y_2}{2}.\\
\end{eqnarray*}
$$
Now the distance of $(x_1,y_1)$ from the line passing through the midpoints is
$$\frac{|(x_1-x_2,y_1-y_2)\times(u,v)|}{2\sqrt{u^2+v^2}}
=\frac{|(x_1-x_2)v-(y_1-y_2)u|}{2\sqrt{u^2+v^2}}.
$$
Naturally, the point $(x_2,y_2)$ is at the same distance from the line, on the other side of it. Similarly, the distance of the endpoints of the other segment from the line is
$$\frac{|(x_3-x_4,y_3-y_4)\times(u,v)|}{2\sqrt{u^2+v^2}}.
$$
I suggest using the sum of the squares of the above distances, scaled to remove the constant factor $1/2$, as your custom distance function for the clustering algorithm. The expressions simplify a bit when you square them:
$$
\frac{((x_1-x_2)v-(y_1-y_2)u)^2+((x_3-x_4)v-(y_3-y_4)u)^2}{u^2+v^2}.
$$
$\frac{\pi}{2}$to display $\frac{\pi}{2}$ is easier to read than pi/2 – Henry Feb 22 '18 at 10:51