Let $X$ be the random number of trials needed to obtain three consecutive successes. Note the following assumption: if there is a failure before the third consecutive successful trial, then we start all over, in which the next trial's probability of success (immediately after the observed failure) is $0.9$. We wish to compute $\operatorname{E}[X]$, the mean number of trials needed. To this end, $$\operatorname{E}[X] = (0.9)(0.8)(0.7)(3) + (0.1)(1 + \operatorname{E}[X]) + (0.9)(0.2)(2 + \operatorname{E}[X]) + (0.9)(0.8)(0.3)(3 + \operatorname{E}[X]),$$ from which it follows that $$\operatorname{E}[X] = \frac{655}{126} \approx 5.19841.$$ At a cost of $100$ per trial this translates to an average cost of $519.841$ which agrees with your simulation.
Why does this formula work, and what does it mean? Well, look at it this way. If the first three trials are successful with probability $(0.9)(0.8)(0.7)$, then it took $3$ trials. Otherwise, the first trial failed with probability $0.1$, in which case we start over and the expected number of additional trials remains the same, so the total expected number of trials after the failure of the first trial is $1+\operatorname{E}[X]$. This explains the first two terms on the right-hand side.
If the first trial is successful but the second fails (and this has a probability of $(0.9)(0.2)$ of occurring), then again we start all over and the expected number of additional trials is again $\operatorname{E}[X]$, but this time we used $2$ trials, so the total expected number of trials after the failure of the first two is $2+\operatorname{E}[X]$.
Finally, if the first two trials are successful but the third fails, then we spent three trials but need to start over.
Notice that $$(0.9)(0.8)(0.7) + (0.1) + (0.9)(0.2) + (0.9)(0.8)(0.3) = 1$$ so we counted all the possible cases.
The model we are actually working with is a discrete-time Markov chain model with transition matrix $$\mathcal P = \begin{bmatrix}0.1 & 0.9 & 0 & 0 \\
0.2 & 0 & 0.8 & 0 \\
0.3 & 0 & 0 & 0.7 \\
0 & 0 & 0 & 1 \end{bmatrix},$$ where each row corresponds to a state of the system in which there are a certain number of consecutive successes. So the first row corresponds to the state where we have no successes; the second row is for when we have had one success; and so forth, until the final row is the state where we have achieved three consecutive successes. The columns represent the state we are transitioning to, from a given row, and the value in a cell represents the probability of transitioning from one state to another. So for example, in the second row, third column, the value $0.8$ represents the conditional probability that, given we have observed one success so far (row 2), we observe another success in the next trial, thus transitioning to row 3. The value $0.3$ in the third row, first column, represents the conditional probability that, given we have observed two consecutive successes, we fail on the third trial and therefore must start over with zero consecutive successes (row 1). The last row and column has a $1$, because it represents the state in which we have obtained three consecutive successes, and therefore stop trying.
By using various properties of Markov chains, we can use this transition matrix $\mathcal P$ to compute the mean time to reach the absorbing state (row 4). We can also find a probability distribution for the time to obtain three consecutive successes, but for this particular matrix, it is a long and convoluted expression. Numeric estimates are provided as follows:
$$\begin{array}{c|c|c}
x & \Pr[X = x] & \Pr[X \le x] \\
\hline
3 & 0.504 & 0.504 \\
4 & 0.0504 & 0.5544 \\
5 & 0.09576 & 0.65016 \\
6 & 0.127512 & 0.777672 \\
7 & 0.0408744 & 0.818546 \\
8 & 0.0477238 & 0.86627 \\
9 & 0.0396724 & 0.905943 \\
10 & 0.0213864 & 0.927329 \\
11 & 0.019588 & 0.946917 \\
12 & 0.0143776 & 0.961294 \\
13 & 0.00958306 & 0.970878 \\
14 & 0.00777728 & 0.978655 \\
15 & 0.00560823 & 0.984263 \\
16 & 0.00403067 & 0.988294 \\
17 & 0.00309244 & 0.991386 \\
18 & 0.00224614 & 0.993632 \\
19 & 0.00165188 & 0.995284 \\
20 & 0.00123746 & 0.996522 \\
\vdots & &
\end{array}$$
