Big Hint
Better notation can make the problem more clear. Let $\langle\cdot,\cdot\rangle_{I}$ denote the standard inner product on $\mathbb{R}^n$, and let $\langle\cdot,\cdot\rangle_{\beta}$ denote the inner product defined by $\beta$. Let $\mathcal{M}_{\beta}$ denote the matrix of the form. Then we know that $\mathcal{M}_{\beta}=P^tP$ for some invertible matrix $P$.
$O(n)$ is usually defined by $O(n)=\{A\in GL(n,\mathbb{R})\,|\, A^tA=I\}$. An equivalent definition is
$$O(n)=\{A\in GL(n,\mathbb{R})\,|\, \langle Av,Aw\rangle_{I}=\langle v,w\rangle_{I}\}.$$
Consider $G'$ defined by
$$G'=\{A\in GL(n,\mathbb{R})\,|\, \langle Av,Aw\rangle_{\mathbb{\beta}}=\langle v,w\rangle_{\beta}\}.$$
Let $A\in O(n)$, and consider $P^{-1}AP\in GL(n,\mathbb{R})$. Then we have
$$\displaystyle\begin{split}\langle(P^{-1}AP)v,P^{-1}AP)w\rangle_{\beta} &=(P^{-1}APv)^{t}P^tP(P^{-1}APw)\\ &=v^tP^tA^t(P^{-1})^tP^tPP^{-1}APw\\
&=v^tP^tA^tAPw\\
&=v^tP^tPw\\
&=\langle v,w\rangle_{\beta}.\end{split}$$
This shows that $P^{-1}AP\in G'$.
Supplementary
- Let $\langle\cdot,\cdot\rangle_{\mathcal{F}}$ be a bilinear form on $\mathbb{R}^n$, and let $\mathcal{M}_{\mathcal{F}}$ be the matrix of the form with respect to a basis $(v_1,\ldots,v_n)$. Then $\langle v,w\rangle_{\mathcal{F}}=v^t\mathcal{M}_{\mathcal{F}}w$ for $v,w\in\mathbb{R}^n$.
- If the basis is changed to $(v'_1,\ldots,v'_n)$, then the matrix of the form is changed to $P^t\mathcal{M}_{\mathcal{F}}P$, where $P$ is the base change matrix with respect to the two basis $(v_1,\ldots,v_n)$ and $(v'_1,\ldots,v'_n)$. Thus, if $\mathcal{M}_{\mathcal{F}}$ represents inner product with respect to some basis, then we know that $\mathcal{M}_{\mathcal{F}}=P^tIP=P^tP$ for some $P$, where $I$ is the identity matrix.