I have the two equations
$$l - 3m + n = 0$$ and
$$l^2 - 5m^2 + n^2 = 0$$
and from here I am lost on how to proceed. I tried substituting l or m or n into the lower equation but keep getting stumped. (Because I get a double variable, i.e. $mn$, $ln$, $lm$ in each case)
The intersection can be a point, a line or two lines because the plane passes through the origin which is the vertex of the (double cone).
Substituting is a good strategy here: from the first equation we have that $z=3y-x$ and plugging it into the second equation we get $$x^2 - 5y^2 + (3y-x)^2 = 0\Leftrightarrow (x-2y)(x-y)=0 \Leftrightarrow x=2y \lor x=y.$$ Hence, letting $t=y$, we have two (parametric) lines: $$l_1:\;t\to (2,1,1)t \quad \text{and}\quad l_2:\;t\to (1,1,2)t.$$
We can also consider this geometrically, in terms of how the plane intersects the nappes of the cone. Since the nappes extend from the origin, through which the plane passes, the intersections of the plane and cone must be degenerate conics, as Robert Z describes. (The familiar "conic sections" are the intersections of a plane which does not pass through the vertex of the (double) cone.)
The symmetry axis of $ \ x^2 - 5y^2 + z^2 \ = \ 0 \ $ is the $ \ y-$axis, so the cross-sections of the cone in the planes $ \ y \ = \ \pm Y \ $ are circles $ \ x^2 + z^2 \ = \ 5Y^2 \ \ . \ $ The intersection of $ \ x - 3y + z \ = \ 0 \ $ with those respective planes are the lines $ \ x + z \ = \ \pm 3Y \ \ . \ $ These curves have "diagonal symmetry" about the plane $ \ x \ = \ z \ \ , \ $ so there will be two symmetrical intersections on each circle.
(For $ \ x \ = \ z \ \ , \ $ we find $ \ 2x^2 \ = \ 5Y^2 \ \Rightarrow \ x \ = \ z \ = \ \sqrt{\frac52 }· Y \ $ on the circle and $ \ 2x \ = \ 3Y \ \Rightarrow \ x \ = \ z \ = \ \frac32 · Y \ \ . \ $ Since $ \ \sqrt{\frac52 } \ > \ \frac32 \ \ , \ $ there are always two diagonally-symmetric intersections on each circular cross-section.* With $ \ x^2 + z^2 \ = \ 5Y^2 \ $ and $ \ (x + y)^2 \ = \ 9Y^2 \ \ , \ $ those intersections lie on the curve $ \ xz \ = \ 2Y^2 \ \ . \ ) $
*From this, we can discern the circumstance in which a plane through the origin would only have a single line of intersection with the cone.
We can then solve for the coordinates of the intersections on each cross-sectional circle: $$ z \ = \ \pm 3Y \ - \ x \ \ \Rightarrow \ \ x·(\pm 3Y \ - \ x) \ = \ 2Y^2 \ \ \Rightarrow \ \ x^2 \ \mp 3Yx \ + \ 2Y^2 \ \ = \ \ 0 $$ $$ \Rightarrow \ \ ( \ x \ \mp Y \ )·( \ x \ \mp 2Y \ ) \ \ = \ \ 0 \ \ . $$ The coordinates of the intersections are thus $$ (x \ , \ y \ , \ z) \ \ = \ \ ( \ \pm Y \ , \ \pm Y \ , \ \pm 2Y \ ) \ \ \text{and} \ \ ( \ \pm 2Y \ , \ \pm Y \ , \ \pm Y \ ) \ \ . \ $$ So the intersections lie on the lines described parametrically in Robert Z's solution. [The graph above presents the cross-section on $ \ y \ = \ +3 \ \ ; \ $ the intersections in $ \ y \ = \ -3 \ $ are found in the diagonally-opposite quadrant.]
