An approach via Fourier transform
The definition that will be used here is
$$
\mathcal{F}f(\xi) = \int_{-\infty}^{\infty} f(x) \, e^{-i \xi x} \, dx,
\quad
f(x) = \frac{1}{2\pi} \int_{-\infty}^{\infty} \mathcal{F}f(\xi) \, e^{i \xi x} \, d\xi
.
$$
The integral
$$
I(x) := \mathcal{P}\int_{-\infty}^{\infty} \frac{e^{-y^2}}{x-y} \mathrm{d}y
$$
can be seen as the convolution of the tempered distribution
$u(x) = \mathcal{P}\frac{1}{x}$
with the Schwartz function
$\varphi(x) = e^{-x^2}$,
i.e. $I = u * \varphi.$
Taking the Fourier transform we then have
$
\mathcal{F}I
= \mathcal{F}(u*\varphi)
= \mathcal{F}u \, \mathcal{F}\varphi
.
$
Now,
$\mathcal{F}u(\xi) = -i\pi \operatorname{sign}(\xi)$ 1
and
$\mathcal{F}\varphi(\xi) = \sqrt{\pi} e^{-\xi^2/4}$, so $\mathcal{F}I(\xi) = -i\pi^{3/2} \operatorname{sign}(\xi) \, e^{-\xi^2/4}$.
Thus,
$$\begin{align}
I(x)
&= \frac{1}{2\pi} \int_{-\infty}^{\infty} -i\pi^{3/2} \operatorname{sign}(\xi) \, e^{-\xi^2/4} e^{i \xi x} \, d\xi \\
&= \frac{\sqrt{\pi}}{2i} \int_{-\infty}^{\infty} \operatorname{sign}(\xi) \, e^{-\xi^2/4} e^{i \xi x} \, d\xi \\
&= \sqrt{\pi} \int_{0}^{\infty} e^{-\xi^2/4} \sin\xi x \, d\xi . \\
\end{align}$$
Taking the derivative gives
$$\begin{align}
I'(x)
&= \sqrt{\pi} \int_{0}^{\infty} e^{-\xi^2/4} \xi \cos\xi x \, d\xi \\
&= \sqrt{\pi} \left( \left[(-2 e^{-\xi^2/4}) \cos\xi x \right]_{0}^{\infty} - \int_{0}^{\infty} (-2 e^{-\xi^2/4}) \, (-x \sin\xi x) \, d\xi \right) \\
&= \sqrt{\pi} \left( 2 - 2 x \int_{0}^{\infty} e^{-\xi^2/4} \sin\xi x \, d\xi \right) \\
&= 2 \sqrt{\pi} - 2 x I(x), \\
\end{align}$$
which is easily solved using integrating factor, giving
$$
I(x) = 2 \sqrt{\pi} \, e^{-x^2} \int_0^x e^{t^2} \, dt = \pi \, e^{-x^2} \operatorname{erfi}(x).
$$