This process you describe is very similar to the Mark and Recapture Technique. In the usual mark and recapture technique however, we recapture a predetermined number of animals in the sample without regard to what we have currently caught (I.e. no waiting until we catch a 10th untagged).
The concept relies on the intuitive idea that the percentage of tagged population should reflect percentage of tagged sample and vice versa. This gives us the formula in your example:
$\frac{\text{recaptured tagged}}{\text{recaptured total}} \approx \frac{\text{total tagged}}{\text{total population}}$. Plugging in values: $\frac{2}{12} \approx \frac{10}{N}$
Solving for $N$ above, we get as you guessed $12\cdot 10/2 = 60 \approx N$
The difficulty in your specific example however, is that we intentionally stop after having found our tenth untagged animal. This overly complicates things, however. I will continue thinking on how to incorporate this information into an answer, however I encountered difficulties in my previous attempts at reaching a solution via explicitly defining a joint probability distribution for the random variables (num caught, and population size). Perhaps someone more capable than I will come around and answer this more complicated version, but for now I will sleep on the thought.