The key is the condition "a fair game if the person plays optimally".
You have analyzed only one strategy: playing the game until you win the money. In the first version of the problem, that is necessarily the optimal strategy: if it was worth playing the game with $4$ boxes, it becomes worth playing the game with $3$ boxes, as well.
In general, we can consider $4$ strategies; the $k^{\text{th}}$ strategy is to stop after $k$ boxes have been opened (if you haven't yet won). In the second version of the problem:
- The $k=1$ strategy can be disregarded: if it was worth opening the first box, it is worth opening the second box, which has the same price but better chances.
- The $k=3$ strategy can be disregarded: if it was worth opening the third box, paying $Z$ for a chance at $\$100$, then it's worth opening the fourth box and paying $Z$ for a certain $\$100$.
But the $k=2$ strategy and $k=4$ strategy are both valid. So the constraints on fairness are that:
- The $k=2$ strategy does not earn free money: $$\frac14(100-X) + \frac14(100-2X) + \frac12(-2X) \le 0.$$ Equivalently, $7X \ge 200$.
- The $k=4$ strategy does not earn free money: $$\frac14(100-X) + \frac14(100-2X) + \frac14(100-2X-Z) + \frac14(100-2X-2Z) \le 0.$$ Equivalently, $7X + 3Z \ge 400$.
- There is a way to break even: equality holds in at least one of the cases above.
I would consider any strategy that satisfies all three conditions "fair, if the person plays optimally". This means that either $X = \frac{200}{7}$ and $Z \ge \frac{200}{3}$, or $X \ge \frac{200}{7}$ and $Z = \frac{400-7X}{3}$. This includes:
- The case $X=40$ and $Z=40$, which is unsurprisingly fair, because it's the same as the first version of the game.
- The case $X=\frac{200}{7}$ and $Z = 100000$, which is fair if the person plays optimally and stops after two boxes. (In this game, it is a bad idea to open the third box, since you pay more money to open it that you could possibly win.)
Possibly the intended solution, though, is $X = \frac{200}{7}$ and $Z = \frac{200}{3}$, which is fair if the person plays either the $k=2$ or the $k=4$ strategy.
Alternatively, you could argue that even if equality does not hold in either case, we have "a fair game if the person plays optimally". The game with $X=100000$ and $Z=100000$ is a fair game if the person plays optimally: if the person walks away and does not play at all, they break even.