The condition that $$\det \begin{bmatrix} x_1 & x_2 \\ y_1 & y_2\end{bmatrix} = 0$$ is equivalent to saying that the vectors $(x_1,x_2)$ and $(y_1,y_2)$ are collinear. You can get this from some linear algebra: if the determinant is $0$, the matrix is not full rank, so its second row is a multiple of the first. But it's probably simpler to just rewrite the condition $x_1 y_2 - x_2 y_1 = 0$ as $\frac{x_1}{x_2} = \frac{y_1}{y_2}$, given that we're told $x_2 \ne 0$ whenever $(x_1,x_2) \in U$.
In that case, it's straightforward to prove transitivity: if $(x_1,x_2)\,R\,(y_1,y_2)$ and $(y_1,y_2)\,R\,(z_1,z_2)$, then $\frac{x_1}{x_2} = \frac{y_1}{y_2}$ and $\frac{y_1}{y_2} = \frac{z_1}{z_2}$, so by transitivity of equality we have $\frac{x_1}{x_2} = \frac{z_1}{z_2}$ which can be rewritten as $(x_1,x_2)\,R\,(z_1,z_2)$.
(Note that we did need $x_2 \ne 0$ in the definition of $U$. If we allow the element $(0,0)$ to be in $U$, then $(x_1,x_2)\,R\,(0,0)\,R\,(y_1,y_2)$ for any $(x_1,x_2)$ and $(y_1,y_2)$, which breaks transitivity.)
It's generally a good idea, when dealing with equivalence relations, to look for a function "characterizing" them: some function $f$ such that $a\,R\,b$ if and only if $f(a) = f(b)$. This makes proving that relation is an equvalence relation a lot easier, and generally gives you helpful insight into how $R$ works.
