So your segment has endpoints $\mathbf p_1, \mathbf p_2$ and one of the planes is $\mathbf n \cdot \mathbf r = d$, where $\mathbf n$ points into the tetrahedron, then examine $\mathbf n \cdot \mathbf r - d$ for $\mathbf r = \mathbf p_1$ and $\mathbf r = \mathbf p_2$.
- If they are both negative for any of the 4 planes, then your segment does not intersect the tetrahedron.
- If for some plane, one is negative and the other point is $0$, and if the $0$ point is negative for some other plane, then your segment also does not intersect the tetrahedron.
- If all 4 planes have both points giving positive results, your segment lies entirely inside the tetrahedron, but does not intersect the faces.
Otherwise, your segment intersects the tetrahedron in at least one point. Go through your planes and if for any of them you have a negative point:
- If the other point is $0$, then the $0$ endpoint is the sole point of intersection with the tetrahedron.
- Otherwise, replace the negative point with $t\mathbf p_1 + (1-t)\mathbf p_2$, where $t = \frac{d - \mathbf n \cdot \mathbf p_2}{\mathbf n \cdot \mathbf p_1 - \mathbf n \cdot \mathbf p_2}$. This is the intersection of the segment with that plane. Check this new point against the all other planes.
When you are done, both points will have positive or 0 values for each plane, and at least one of them will have 0 values for some plane.
- If a point gives a $0$ value for some plane, then it lies within the triangular face in that plane.
- If it is $0$ for 2 planes, it lies within their common edge.
- If it is $0$ for 3 planes, it is their common vertex.
- If it is $0$ for all 4 planes, then your "tetrahedron" is degenerate.
- If it is positive for all planes, then it lies within the interior of the tetrahedron, and only the other endpoint is a point of intersection.
The segment is in a plane if both endpoints lie within that plane. The segment is along an edge, if both endpoints lie within the edge.