A general technique to solve such problems where we have infinitely many independent trials and two disjoint events $A$ and $B$, where each trial we have $P(A)=p$, and $P(B)=q$ with $p+q \le 1$. Let us denote the chance that doing trials until $A$ or $B$ occurs, we stop at event $A$ without having seen $B$.
Then conditioning on the outcome of the first trial $X_1$:
$$x = P(A \text{ before } B| X_1 \in A)P(X_1 \in A) + P(A \text{ before } B| X_1 \in B)P(X_1 \in B) + P(A \text{ before } B| X_1 \notin A \cup B)P(X_1 \notin A\cup B)$$
We can use that $P(A \text{ before } B| X_1 \in A) = 1$ (we immediately have $A$ happening so we "win") and $P(A \text{ before } B| X_1 \in B)P(X_1 \in B) = 0$ (we have event $B$ so we stop and "lose") and by definition $P(X_1 \in A) = p$, $P(X_1 \in B)=q$ and $P(X_1 \notin A \cup B)$ means that neither $A$ nor $B$ occurred, and this happens with chance $1-p-q$. Also the probability $P(A \text{ before } B| X_1 \notin A \cup B)$ is just $x$ again, because if the first trial gives no stop condition $A$ or $B$, it's just the same situation as we had before the very first trial, and so the same probability that $A$ occurs before $B$.
So $$x=p\cdot 1 + q\cdot 0+x(1-p-q)$$ or $$(p+q)x =p$$ or $$x=\frac{p}{p+q}$$
Similarly, the probability that $B$ occurs before $A$ is $\frac{q}{p+q}$, together $1$.
In your case $A$ is throwing a $5$ which has probability $\frac{4}{36}$ and $B$ is throwing $8$ which has probability $\frac{5}{36}$.
The formula then gives $\frac{\frac{4}{36}}{\frac{4}{36} + \frac{5}{36}} = \frac{4}{9}$ as it should.