This is a good question that really messes with your intuition, and it's a subtle one. You have the basic idea though. Hopefully I can elaborate in a way that might help it make sense.
$\mu$ for the population either is or is not in the interval - it's a binary thing, there is no "chance" of it being anywhere. On the other hand, what this $95\%$ confidence basically means is that, if you were to generate many, many samples each with their own mean, and used those sample means $\bar x$ to generate likewise $95\%$ confidence intervals, about $95\%$ of those generated intervals would in fact contain the population mean $\mu$.
For any particular interval, $\mu$ is either in it or it's not. There is no chance of it, it's only a single interval after all. But if you were to generate many confidence intervals, on average, you would expect about $95\%$ of them to contain $\mu$.
Or phrased differently yet again, from a huge bunch of confidence intervals generated this way, you could uniformly randomly pick one interval containing $\mu$ with $95\%$ probability. The chosen interval either does or doesn't. But you'd have an about $95\%$ chance at picking such an interval.
Minor Footnote:
We say "about $95\%$" - instead of literally or exactly that - because while we might deviate from that a bit in obtaining finitely many samples, the idea is that the proportion of how many we create that contain $\mu$ will approach $95\%$ as we create more and more and more samples. After all, you couldn't get $95\%$ exactly even if you had $999,999$ samples, for instance, the math just doesn't work - but you can get absurdly close. $96/100$ samples having $\mu$ doesn't mean there's some flaw, it rectifies itself as you generate more samples.
Going deeper into this phenomenon and other ways you might see it arise in mathematics, and how we might make this idea rigorous, is a topic for another question though. (One could even say it's obvious but I feel it's important enough to be worth noting.)
