Suppose that we have a variational problem,
$\int_{t_1}^{t_2}f(\vec{x}(t),\vec{x}'(t),|\vec{x}(t)|)dt$
subject to the constraint:
$|\vec{x}|=1$
where $\vec{x}(t)=\left\{x_1(t),x_2(t),x_3(t) \right\}.$ Are we allowed to rewrite the integral as
$\int_{t_1}^{t_2}f(\vec{x}(t),\vec{{x}'}(t),1)dt$
and continue with the conventional variational calculus procedure with Lagrange multiplier (i.e., substituting the constraint before taking the variational derivatives) ?
Yes. Let $\Phi(x)=\int_{t_1}^{t_2}f(\vec{x}(t),\vec{x}'(t),|\vec{x}(t)|)\,dt$ and $\Psi(x)=\int_{t_1}^{t_2}f(\vec{x}(t),\vec{x}'(t),1)\,dt$. You are looking for $\inf_{x\in A} \Phi(x)$ where $A$ is the set of admissible functions $x$. The following claim hardly needs a proof: it is a tautology.
Claim: If $\Phi(x)=\Psi(x)$ for all $x\in A$, then $\inf_{A} \Phi=\inf_{A} \Psi$. Moreover, the set of extremals is the same: $\{x\in A: \Phi(x)=\inf_A \Phi\}$ is the same set as $\{x\in A: \Psi(x)=\inf_A \Psi\}$.
Therefore, you can work with $\Psi$ instead of $\Phi$.