Your confusion is a common one. You haven't realized that "closure" is a “relative” concept. This means that it's not quite correct to talk about ‘the’ closure of a set $A$. Rather, $A$ has a closure when considered as a subset of some larger space $X$, and the closure of $A$ might be different depending on what $X$ is.
This is similar to how the idea of a set complement is relative. What's the complement of the set of even integers? Well, if you consider it to be a subset of $\Bbb Z$, then its complement is the set of odd integers. But if you consider it to be a subset of $\Bbb R$, the complement includes not just the odd integers but also non-integers like $\frac12$.
Context matters, and when we say "the complement of $A$” we depend on the listener to recognize the context. Similarly when we say “the closure of $A$”.
The notation $\bar A$ is misleading, because what $\bar A$ is will depend on $X$, but the $X$ doesn't appear in the notation.
Here's Mendelson's definition of “closure” (Definition 4.2, page 124):
Let $A$ be a subset of a topological space. A point $x$ is said to be in the closure of $A$ if, for each neighborhood $N$ of $x$, $N\cap A\ne \emptyset$.
This depends on what the neighborhoods of the enclosing space are. Mendelson has obscured this by not even naming the enclosing space, which I have called $X$. He implies, but doesn't say, that the points $x$ are points of the space $X$.
I'm going to rephrase Mendelson's definition to make this explicit:
The closure of a subset $A$ of a topological space $X$ is the set of all points $x\in X$ such that, for every open set $N$ of $X$ containing $x$, $$N\cap A\ne\emptyset.$$
Now let's do the example you asked about. Let $A=(0,1)$ and $X =\Bbb R$ with the usual topology. The closure of $A$ in $\Bbb R$ is $[0,1]$ as I think you know.
But now let $A=(0,1)$ and $X = \Bbb R^+ = (0, \infty)$, again with the usual topology. The closure of $A$ in $\Bbb R^+$ is $(0, 1]$.
Why is the point $0$ missing from $\bar A$? Because the definition says
The closure… is the set of all points $x\in X$ such that…
and if $X=\Bbb R^+$, then $0\notin X$, so $0$ can't be in the closure of $A$. It's not in the closure because it's not part of the universe!
Similarly, you can consider $X = (0,1)$ with the usual topology. The closure of $A=(0,1)$ in this space is $(0,1)$. It doesn't include points $0$ or $1$ because those are not part of the space at all.
And this is the example you're asking about: If $(0,1)$ is the whole space, then its closure is $(0,1)$.