$BP \parallel AF \implies ∠PBF= ∠BFA=45° \text{ (alternate interior angles)}$
$ \implies ∠BAF=∠BFA \text{ (sum of angles in triangle)} $
$\implies AB=FB \text{ (sides opposite to equal angles in a triangle)}$
$\therefore BC= CF+FB=a+AB \implies AB=BC-a$
Using this, in the equation you got
$(BP - \frac1{ \sqrt{2}}BC)^2 - \frac{AB^2}2 =0 \implies (BP - \frac1{ \sqrt{2}}BC)^2 - (\frac{BC-a}{\sqrt{2}})^2=0$
$\implies \{(BP - \frac1{ \sqrt{2}}BC)+(\frac{BC-a}{\sqrt{2}})\}\{(BP - \frac1{ \sqrt{2}}BC)-(\frac{BC-a}{\sqrt{2}})\}=0$
$\implies (BP - \frac{a}{ \sqrt{2}})(BP - \frac2{ \sqrt{2}}BC+\frac{a}{ \sqrt{2}})=0$
$\implies BP=\frac{a}{ \sqrt{2}} \text{ or }BP=\sqrt{2} BC-\frac{a}{ \sqrt{2}} \tag{i} \label{i}$
Let us assume $BP=\sqrt{2} BC-\frac{a}{ \sqrt{2}} \tag{ii}$
$$\text{In }\triangle ABF, AB+BF>AF \implies 2(BC-a)>AF (\because AB=BF=BC-a$$
$$\implies 2BC-2a+a>AF+a>AC (\because AF+a>AC \text{ in } \triangle AFC$$
$$\implies \sqrt{2} BC-\frac{a}{ \sqrt{2}}>\frac{AC}{\sqrt{2}} \text{
(divided both sides by }\sqrt{2})$$
$$\implies BP>\frac{AC}{\sqrt{2}}=\frac{2R}{\sqrt{2}}=\sqrt{2}R\text{ (using (ii))}$$
$\implies BP>AP \implies \overset\frown{BP}>\overset\frown{AP}$ which is a contradiction. Therefore our assumption (ii) is wrong.
$\therefore BP\neq \sqrt{2} BC-\frac{a}{ \sqrt{2}}$ and (i) $\implies BP =\frac{a}{ \sqrt{2}} $
$\textbf{Method 2: Using similarity of triangles}$
$\begin{array}{l}
\text{In } \triangle ABP \text{ and } \triangle AFC\\
\angle ABP= \angle AFC =135° (\because \angle AFC=180°-\angle BFA)\\
\angle BAP=\angle FAC (\because \angle BAP=\angle BAC - \angle PAC =\angle BAC - 45°=\angle BAC - \angle BAF=\angle FAC)
\end{array}$
$\therefore \triangle ABP \sim \triangle AFC$ by AAA similarity criterion.
$\implies \frac{BP}{FC}=\frac{AP}{AC} \implies \frac{BP}{a}=\frac{\sqrt{2} R}{2R}=\frac1{\sqrt{2}}$
$\therefore BP=\frac{a}{\sqrt{2}}$