In the following, I try to answer just this part of the question:
"How did they straightaway know this?"
In the usual definition of polar coordinates, as you know,
the relationship between the Cartesian coordinates of a point,
$(x,y)$, and the polar coordinates $(r,\theta)$ of the same point is
$$ x = r \cos \theta,$$
$$ y = r \sin \theta.$$
This means that if you have an equation that is satisfied
by the Cartesian coordinates of a particular point, $(x,y)$,
you can simply replace $x$ by $r\cos\theta$ and replace
$y$ by $r \sin \theta$ in that equation, and you will have an
equation satisfied by the polar coordinates of that point.
This is true because in the descriptions of this point by
its coordinates, $x$ and $r\cos\theta$ are the same number,
likewise $y$ and $r \sin \theta$.
There is a particular curve in the plane, such that every point on
that curve satisfies the given equation,
$$x^2 + y^2 = 2x.$$
Replacing $x$ by $r\cos\theta$ and $y$ by $r \sin \theta$, we
get another equation that must be true for every point on that curve:
$$(r\cos\theta)^2 + (r\sin\theta)^2 = 2(r\cos\theta).$$
Since $\cos^2\theta + \sin^2\theta = 1$, the left side of this equation
reduces to $r^2$. That reduction happens so often in coordinate geometry
that most people who see $x^2 + y^2$ will immediately replace it by $r^2$.
So you don't see the equation above written out in full,
but instead simply write $r^2 = 2(r\cos\theta).$
One might well ask not just how "they" knew they could do this, but
also why it might have occurred to them to try it.
As with a lot of math problems, there are a couple of likely explanations:
(1) you tried a bunch of things and this is the one that worked
(and you threw out your notes on all the failed methods);
(2) you were just lucky this was the first thing you tried; or
(3) something in the problem reminded you of something you've seen before
that works out nicely when you do this change of coordinates.
By the time someone gets around to writing this in a textbook or class notes,
it's probably due to the last reason, but someone seeing this for the first
time only has the first two options.
I think you can legitimately construct a system of polar coordinates in which
$r \cos \theta = x - 1$ instead of the usual $x$, but it's not the "obvious"
coordinate conversion and apparently it didn't help much in this problem.