Hard to imagine. However we can imagine living as inhabitants of a 2D world, which is embedded in a 3D world. That 2D world could be the screen you are looking at.
Say I have a cup of water sitting on a table. Is it safe to say that
the cup, the table and me are all in the same position in the 4th
dimension?
Well in my reduced example, you would see an image of a cup, table and yourself on the screen. Now all the objects have 3D coordinates $(x, y, z)$,
where $x$ and $y$ might be horizontal and vertical positions on the screen and $z$ is orthogonal to the screen, e.g. it could be the distance from the screen. All the mentioned objects share the same $z$-coordinate $z = 0$, as they stay on the screen.
Secondly, if I began moving the cup into the 4th dimension, what would
I see? Would it start to fall through the table? Would my hand be able
to go through it?
In the reduced example:
The question is would all or only part of the cup move into the third dimension?

The cup is flat (*). You could move that flat cup all to $z = 1$ then it would hover in front of the screen, above it, parallel to it.
You, in a similar situation like ours, would only be able to perceive the 2D world, what is on the screen. The cup would have vanished from your view.

If you just rotate the cup into the 3D world you would have a part above the screen, a part below the screen and the rotational axis part stays with $z=0$. The 2D cup would shrink to a 1D line in your view.
Update: (*) Is it really flat? What if it wasn't flat the whole time? But for some reason we can only see and interact with that 2D-slice which is part of our 2D world. So it might change shape too.
Or would it disappear instantly when it moved any amount in the 4th
dimension?
Only if all of the points that it is made up, get $z \ne 0$.
Similarly, when it came back.. if I had my hand in the way, would I
feel it push against my hand? Would it push my hand out of the way?
Could it expand within my hand from the 4th dimension or would it
simply not reach because I was in the way?
No idea how physical forces would act. It might squeeze you out in the third dimension.