You are correct to think of $\frac{1}{3}$ of $\frac{9}{10}$. Let's go through the problem and calculate the actual fraction of participants in each cutoff. First we are given $\frac{1}{10}$ finish in the first $3$ hours. Then $\frac{1}{3}$ of the remainder finished in under $5$ hours - that is $\frac{1}{3}\times\frac{9}{10}=\frac{3}{10}$ finished in this period. Finally, twice as many people finished in the second to last group as in the last group. What is that in a fraction form? Let's call the number who finished in the last group $x$. We have $1-\frac{1}{10}-\frac{3}{10}=\frac{6}{10}=\frac{3}{5}$ of the participants remaining. Hence to find $x$, we just need to solve $x+2x=\frac{3}{5}$, so we have $x=\frac{1}{5}$. This leads to the conclusion that $650$ was $\frac{1}{5}$ of the participants, so the total number of participants was $650\times 5=3250$.
Alternatively, we could proceed from where you left off - having found $650$ in the last group and $1300$ in the second to last group. We know that $\frac{1}{10}$ of the total were in the first group and then $\frac{1}{3}\times\frac{9}{10}=\frac{3}{10}$ were in the second group. So in the first two groups combined, there were $\frac{1}{10}+\frac{3}{10}=\frac{4}{10}=\frac{2}{5}$ of the participants. Hence the total number of people you have found so far account for the other $\frac{3}{5}$ of participants. Hence the total number of participants would be $(650+1300)\times\frac{5}{3}=3250$.
