Typically, the only thing that matters are the random variables and the assumptions on those variables. The underlying space just needs to be "sufficiently rich" to support those variables.
When I was learning probability, I was given a problem in which Alice and Bob play a game rolling a die over and over again. The winner is the fist person to get a $6$. What is the probability that Alice wins? The problem wasn't formalized as a probability space, and this got me worried. How do I know I'll pick "the right" space?
One way to model it would be to take the space $\{1, ... 6\}^\mathbb N$ of all sequences of numbers in $\{1, ... 6\}$. Another way would be to just take the space of all finite sequences of such numbers, in which the last digit is $6$. Equivalently, we could take the set of positive integers divisible by $6$ (which we think of as representing a sequence of die rolls by way of base-6 notation, you see). Okay, fine, presumably we'll get the same answer to the question no matter which space we use. But... how do I know for sure we'll get the same answer? What if I pick a space arbitrarily and get one answer, but then somewhere out there there's another, equally plausible model which would have yielded another answer?
Worse than that: even ignoring models that I didn't think of, how do I even know the models I did think of are good models for the problem? Say I choose to model the problem using the space of all sequences of numbers in $\{1, ... 6\}$. Now I need a sigma algebra and a probability distribution. Well, I know that for example $P(\{6\}\times\{1, ... 6\}^\mathbb N)=\frac 1 6$, right? But hang on. I can't just say I know that. For god's sake, these probabilities are the things I'm trying to calculate in the first place, I can't just say "oh, I know intuitively they're going to be such and such". Otherwise I might as well just say "I know intuitively the answer to the question is 0.5" (it isn't) and call it a day!
Is the situation hopeless? Well, no. I eventually understood how it is that you can resolve these doubts. We have to fall back to the oldest tool in mathematics: the axiomatic method. We may not know in advance the exact probability distribution governing the outcome of this dice game, but we do know that it satisfies certain axioms. Just like Euclid doesn't care exactly what "points" and "lines" are as long as they satisfy his axioms, we don't need to worry about anything other than our random variables and the axioms they satisfy:
- There is a sequence of die rolls $X_1, X_2, ...$.
- Each one is uniformly distributed on $\{1, ... 6\}$.
- They are all independent from one another.
This is how we get out of the dead-end of just having to say "I intuitively know the probability distribution is going to be such and such". We don't need to make infinitely many implicit intuitive assumptions about the probability distribution, we just need to make three explicit assumptions, and reason from those. Just like in geometry: rather than reasoning from a swamp of implicit "visual intuition", we reason from a finite list of explicit assumptions.
This also resolves the problem of having to worry about weird models of the problem that we may not have thought of but which would yield different answers. I don't have to just say "well, every space that models the problem that I can think of yields the same answer". I know that every space that models the problem will yield the same answer, because "models the problem" means "satisfies these axioms", and I only used those axioms in my reasoning.
Thus, not only are we permitted to not explicitly state the underlying space, but doing so is one of the key ideas that allows us to be rigorous in probability theory.