For simplicity, I will assume that the center of $R$ is the origin, that is, the intersection
of the two axes with which $R$ is aligned, which I will call the $x$ and $y$ axes.
To apply the calculations to an axis-aligned rectangle in arbitrary position,
translate it to the origin, find $P$, and translate everything back.
If the rotation angle is an exact integer multiple of $\frac\pi2,$
the desired rectangle $P$ is simply the rectangle $R'$ itself.
Otherwise, let $\theta$ be the reference angle of the angle of rotation of $R'.$
That is, $\theta$ is the acute angle made by the rotation angle and the $x$-axis.
Therefore $0 \lt \theta \lt \frac\pi2.$
Let $h$ be the height of the rectangle (maximum difference in $y$ coordinates),
$w$ the width (maximum difference in $x$ coordinates).
Then if $h = w,$ the vertices of $P$ have coordinates
$$
\left(\pm\frac{h}{2 (\sin\theta + \cos\theta)}, \pm\frac{h}{2 (\sin\theta + \cos\theta)}\right).
$$
If $h \lt w$ and $\sin(2\theta) \geq \frac hw,$ the vertices of $P$ have coordinates
$$ \left(\pm\frac{h}{4\sin\theta}, \pm\frac{h}{4\cos\theta}\right). $$
If $h \gt w$ and $\sin(2\theta) \geq \frac wh,$ the vertices of $P$ have coordinates
$$ \left(\pm\frac{w}{4\cos\theta}, \pm\frac{w}{4\sin\theta}\right). $$
Otherwise, the vertices of $P$ have coordinates
$$
\left(\pm\frac{w \cos\theta - h \sin\theta}{2\cos(2\theta)},
\pm\frac{h \cos\theta - w \sin\theta}{2\cos(2\theta)}\right).
$$
That is the entire algorithm, although it can be written differently.
If this algorithm is implemented on a computer,
it may be advisable to rearrange the formulas in the "otherwise" case to
avoid catastrophic cancellation in the numerator when
$\cos\theta \approx \sin\theta$ and $h \approx w.$
To explain the formulas above,
let the vertices of $R'$ be $A, B, C,$ and $D$ in sequence around the perimeter of the rectangle.
Here are some observations:
$P$ fits between the lines $AB$ and $CD$ (the lines that extend two opposite sides of the rectangle) and therefore
$P$ cannot be larger in area than the largest rectangle that fits between the lines $AB$ and $CD.$
For similar reasons, $P$ cannot be larger in area than the largest rectangle that fits between the lines $AD$ and $BC$ (the lines that extend the other two opposite sides of the rectangle).
$P$ must touch at least one side (otherwise it could be made larger in area).
Since it touches one side, by symmetry it touches the opposite side.
If you reflect the figure consisting of $P$ and $R'$ around the $y$-axis,
you get $P$ again, contained in a rectangle $R''$ which is a reflection of $R'$
across the $y$-axis. $R''$ is the result of rotating the rectangle $R$ by the angle $-\theta.$
(The $x$-axis would also work as the line of reflection.)
The vertices of $R'$ and $R''$ all lie on the circumcircle of $R.$
Due to the symmetry of spacing of the vertices around that circle, either $R'$ and $R''$ are the same rectangle, or their vertices alternate around the circle, or their vertices occur in alternating pairs (two from $R'$, then two from $R''$).
In the case of alternating vertices, each side of $R'$ intersects $R''$ twice.
In the case of alternating pairs of vertices, the two longer sides of $R'$ intersect $R''$ twice each and the shorter sides of $R'$ lie completely outside $R''.$
If $R'$ and $R''$ are not the same rectangle, by symmetry four of their intersection points are on the $x$ and $y$ axes. If there are more intersection points, the remaining intersections are the vertices of a rectangle centered at the origin and aligned with the axes.
If $R'$ and $R''$ are not the same rectangle, then wherever $P$ touches a side of $R'$, it must do so at or between the two points where $R''$ intersects that side.
(In the case of alternating vertices, two sides of $R'$ have no intersections with $R''$, but $P$ cannot touch those sides.)
Let $M$ be the midpoint between the two points of intersection of the line $AB$ with the $x$ and $y$ axes.
Then the largest-area rectangle that fits between the lines $AB$ and $CD$ is a rectangle with a vertex at $M.$
Moreover, if $P$ touches the line $AB$ at a point other than $M,$ the farther that vertex of $P$ is from $M,$ the smaller in area $P$ is.
(There are various ways to prove this.)
Likewise, if $N$ is the midpoint between the two points of intersection of the line $AD$ with the $x$ and $y$ axes,
the largest-area rectangle that fits between the lines $AD$ and $BC$ is a rectangle with a vertex at $N.$
If $P$ touches the line $AC$ at a point other than $N,$ the farther that vertex of $P$ is from $N,$ the smaller in area $P$ is.
If side $AD$ is shorter than side $AB,$ the point $N$ on the line $AD$ will never lie inside $R''$
and therefore $N$ can never be a vertex of $P$.
So if $P$ touches side $AD,$ it must do so at the intersection of $R''$ with $AD$ closest to $N.$
But either the point $M$ on line $AB$ lies between the two intersections of $R''$ with side $AB,$
or one of those two intersections lies between $M$ and the other intersection (which is on an axis).
So in the case where $R$ is not a square, $P$ is a rectangle with a vertex at one of the following two points:
- The midpoint of the two points where an (extended) longer side of $R'$ intersects the $x$ and $y$ axes,
lying on that (unextended) side of $R'$ between two intersections of that side and $R''.$
- An intersection of $R'$ and $R''$ that is not on an axis, where the distance between the two
intersections of $R''$ with the same longer side of $R'$ is less than half the distance between
the two axis-intercepts of that (extended) side.
But if $R$ is a square, the vertices of $R'$ an $R''$ always alternate around the circle
and the vertices of $P$ are always at the intersections of $R'$ and $R''$ that are not on the axes.
In the figure below, $R'$ is shown with sides $AB$ and $AD$ extended;
$R''$ is the rectangle $A'B'C'D'$;
the green rectangle is a rectangle with vertex $M,$
the blue rectangle is a rectangle with vertex $N,$
and the red rectangle is the rectangle with vertices at intersections of $R'$ and $R''.$
Assuming the rotation angle is measured in a counterclockwise direction,
$\theta$ is about $23$ degrees (or an equivalent angle) in this figure.
Because $M$ is the midpoint of the hypotenuse $EF$ of right triangle $\triangle EOF,$
the triangles $\triangle EMO$ and $\triangle FMO$ are always isosceles.
The base angle of $\triangle FMO$ is $\theta,$
and the line $MO$ is a centerline of rectangle $R''$ parallel to edge $A'B'.$
I have omitted the proofs for some of these facts in order to keep an already-too-long answer from becoming even longer, but I have created an
interactive demonstration
in which you can adjust the height and width of $R$, rotate $R'$ by moving vertex $A$, and observe the extensions of sides $AB$ and $AD$ and the various rectangles that might be $P.$
Now suppose $R$ has height $h > 0$ and width $w > 0$ so that one vertex is at
$\left(\frac w2,\frac h2\right)$ and suppose $w > h.$
Label the vertices of $R'$ in counterclockwise order so that
$A=\left(\frac w2,\frac h2\right)$ and
$D=\left(\frac w2,-\frac h2\right)$ when $\theta = 0.$
Then the equation of line $AB$ is
$$ -\left(\frac 2h \sin\theta\right) x + \left(\frac 2h \cos\theta\right) y = 1. \tag1 $$
The equation of line $AD$ is
$$ \left(\frac 2w \cos\theta\right) x + \left(\frac 2w \sin\theta\right) y = 1. \tag2 $$
If $A'B'$ and $C'D'$ are the reflections of $AB$ and $CD$ across the $y$-axis,
then the equation of $A'B'$ is
$$ \left(\frac 2h \sin\theta\right) x + \left(\frac 2h \cos\theta\right) y = 1 \tag3 $$
and the equation of $C'D'$ is
$$ -\left(\frac 2h \sin\theta\right) x - \left(\frac 2h \cos\theta\right) y = 1. \tag4 $$
Equation $(1)$ has $x$- and $y$-intercepts at
$$ \left(-\frac{h}{2\sin\theta}, 0\right) \quad \text{and} \quad
\left(0, \frac{h}{2\cos\theta}\right). $$
The coordinates of the point $M$ (the midpoint of those two axis intercepts) are therefore
$$ \left(-\frac{h}{4\sin\theta}, \frac{h}{4\cos\theta}\right). \tag5 $$
Solving Equations $(2)$ and $(3)$ simultaneously for $x$ and $y,$ we find
the intersection of $AD$ and $A'B'$ at
$$\begin{align}
x &= \frac{w \cos\theta - h \sin\theta}{2\cos^2\theta - 2\sin^2\theta}, &
y &= \frac{h \cos\theta - w \sin\theta}{2\cos^2\theta - 2\sin^2\theta},
\end{align}$$
which can also be written as
$$\begin{align}
x &= \frac{w \cos\theta - h \sin\theta}{2\cos(2\theta)}, &
y &= \frac{h \cos\theta - w \sin\theta}{2\cos(2\theta)}. \tag6
\end{align}$$
Solving Equations $(2)$ and $(4)$ simultaneously for $x$ and $y,$ we find
the intersection of $AD$ and $C'D'$ at
$$\begin{align}
x &= \frac{w \cos\theta + h \sin\theta}{2\cos(2\theta)}, &
y &= \frac{-h \cos\theta - w \sin\theta}{2\cos(2\theta)}. \tag7
\end{align}$$
To decide which formulas to use for which values of the angle $\theta,$
we need to determine when the rectangle $P$ has a vertex at $M$ and when
it has vertices at intersections of $R'$ and $R''.$
The figure below shows a case where $M$ coincides with an intersection of $R'$ and $R'',$
that is, where the green and red rectangles in the figure above are the same rectangle.
Note that this can occur only if $w \geq h.$
Since the distance $OM$ is exactly $w/2$ and
the altitude of $E$ above $OM$ is always $h/2,$
in this case $\sin(\angle EMO) = h/w.$
Since $\theta = \frac12\angle EMO$ in this case, we have $\theta = \frac12\arcsin(h/w).$
But there are many angles of rotation at which the green and red rectangles coincide.
Let $\theta_0 = \frac12\arcsin(h/w)$ (that is, $\theta_0$ is the rotation angle
shown in the figure above).
By inspection of the figure under different rotations, the rotation angles
at which the green and red rectangles coincide are
$n\frac\pi2 \pm \theta_0$ where $n$ is any integer.
That is, the centerline of $R'$ can make an angle $\theta_0$
in either direction from either end of either the $x$- or $y$-axis.
When the centerline of $R'$ makes a smaller angle with either axis,
that is, when $n\frac\pi2 - \theta_0 \lt \theta \lt n\frac\pi2 + \theta_0$
for some integer $n,$
the green rectangle lies partly outside $R'$ and therefore $P$ must be the red rectangle.
When the smallest angle between centerline of $R'$ and either axis
is greater than $\theta_0,$ however, the green rectangle lies inside $R'$
and $P$ must have a vertex at $M$.
One way to determine which rectangle is $P$ for a given rotation angle $\theta$
is to subtract the nearest multiple of $\frac\pi2$ from $\theta$
and compare the absolute value of the result with $\theta_0.$
But if you do not want to do that, there are other formulas.
For example, $P$ has a vertex at $M$ if and only if
$$
\lvert \sin(2\theta) \rvert \geq \frac hw. \tag8
$$
Equivalently, $P$ has a vertex at $M$ if and only if
$$
\cos^2(2\theta) \leq 1 - \frac{h^2}{w^2},
$$
$P$ has a vertex at $M$ if and only if
$$
\frac12 - \frac12\sqrt{1 - \frac{h^2}{w^2}} \leq \cos^2\theta
\leq \frac12 + \frac12\sqrt{1 - \frac{h^2}{w^2}},
$$
and $P$ has a vertex at $M$ if and only if
$$
\frac12 - \frac12\sqrt{1 - \frac{h^2}{w^2}} \leq \sin^2\theta
\leq \frac12 + \frac12\sqrt{1 - \frac{h^2}{w^2}}.
$$
When $P$ has a vertex at $M$ we can use the coordinates given in Formula $(5)$
for that vertex of $P$; but to get all four vertices we reflect across the
$x$- and/or $y$-axis, so the coordinates are
$$ \left(\pm\frac{h}{4\sin\theta}, \pm\frac{h}{4\cos\theta}\right). $$
In the case where Inequality $(8)$ is false,
$P$ must have vertices at the intersections of $R'$ and $R'',$
but then the choice of a formula for the
coordinates depends on whether $AD$ intersects $A'B'$ or $C'D'.$
Inspection of a few cases shows that this depends simply on the quadrant in
which the rotation angle $\theta$ lies:
the intersection is with $A'B'$ (described by Formula $(6)$) if $\theta$
is in the first or third quadrant, $C'D'$ (described by Formula $(7)$) otherwise.
A way to make the decision in this case is:
use Formula $(6)$ if $\sin\theta$ and $\cos\theta$ have the same sign,
so the four vertices of $P$ are
$$
\left(\pm\frac{w \cos\theta - h \sin\theta}{2\cos(2\theta)},
\pm\frac{h \cos\theta - w \sin\theta}{2\cos(2\theta)}\right),
$$
but use Formula $(7)$ if $\sin\theta$ and $\cos\theta$ have opposite signs,
so the four vertices of $P$ are
$$
\left(\pm\frac{w \cos\theta + h \sin\theta}{2\cos(2\theta)},
\pm\frac{h \cos\theta + w \sin\theta}{2\cos(2\theta)}\right).
$$
But if we are willing to apply some absolute values, due to the possible signs
of $\sin\theta$ and $\cos\theta$ in each formula the two formulas above
can be combined as follows: the four vertices of $P$ are
$$
\left(\pm\frac{w \lvert\cos\theta\rvert - h \lvert \sin\theta\rvert}{2\cos(2\theta)},
\pm\frac{h \lvert\cos\theta\rvert - w \lvert\sin\theta\rvert}{2\cos(2\theta)}\right),
$$
If we suppose instead that $h > w,$
we can compute a new set of formulas for the vertices of $P$
using methods similar to what we used above.
But another approach is to reflect the entire problem across the line $y = x$
by swapping $h$ and $w$ and by replacing $\theta$ by $-\theta.$
We can then solve for $P$ using the method already shown;
we can even change $-\theta$ back to $\theta$ after observing that any two
rotations by $\theta$ and $-\theta$ always produce the same rectangle $P.$
Once we have found the coordinates of $P$ in the reflected problem,
we transform it to a solution of the original problem by reflecting it again,
that is, simply by exchanging the $x$ and $y$ coordinates in the solution.
In the case where $h = w,$ the rectangle $R'$ is a square and the rectangle $P$ is also a square.
Formula $(5)$ will apply only when $\cos^2\theta = \frac12,$
but for simplicity we can simply recognize that $P$ is always the red rectangle
and so when $\sin\theta$ and $\cos\theta$ have the same sign
(when $\theta$ is in the first or third quadrant)
we solve Equations $(2)$ and $(3)$ simultaneously so that the coordinates of $P$ are
$$
\left(\pm\frac{h}{2 (\sin\theta + \cos\theta)}, \pm\frac{h}{2 (\sin\theta + \cos\theta)}\right),
$$
but if $\sin\theta$ and $\cos\theta$ have opposite signs
we solve Equations $(2)$ and $(4)$ simultaneously,
so the four vertices of $P$ are
$$
\left(\pm\frac{h}{2 (\sin\theta - \cos\theta)}, \pm\frac{h}{2 (\sin\theta - \cos\theta)}\right).
$$
As in the case where $h \neq w,$ however, we can combine the
same-sign and opposite-sign cases to produce one formula for the vertices of $P$:
$$
\left(\pm\frac{h}{2 (\lvert\sin\theta\rvert + \lvert\cos\theta\rvert)},
\pm\frac{h}{2 (\lvert\sin\theta\rvert + \lvert\cos\theta\rvert)}\right),
$$
Also note that by symmetry, if the rotation angle $\theta$ results in a
particular instance of the rectangle $P,$ then the rotations angles
$-\theta,$ $\pi - \theta,$ and $\pi + \theta$ all result in the same rectangle $P.$
That is, $P$ is determined by the reference angle of $\theta$.
We can find the reference angle of $\theta$ by adding an integer multiple
of $\pi$ to $\theta$ in order to obtain a value in the interval $[0, \pi)$
(that is, find the smallest non-negative value of $\theta$ modulo $\pi$),
and if the result is greater than $\frac\pi2,$ subtracting it from $\pi.$
The final result is an angle in the interval $\left[0, \frac\pi2\right].$
We can therefore set $\theta$ to the reference angle of the actual angle of rotation
and use that value of $\theta$ in any of the preceding formulas.
This simplifies some formulas because $\sin\theta,$ $\cos\theta,$
and $\sin(2\theta)$ are then all non-negative.
I used this idea when writing the algorithm at the start of this answer.
I started writing this answer due to the fact that the previously-posted
answers here do not actually return a rectangle of maximal area in all cases.
However, I later found that algorithms equivalent to mine (or practically equivalent) had long ago been posted on Stackoverflow
here,
here,
and here.
But I hope this answer at least provides some useful information
about the justification of those algorithms.
