By using change of variable, $$x+y=(\surd2)u \text { and } y-x=(\surd2)v$$
Evaluate $$I=\iint(y-x)^2e^{-(x+y)^2}dv\,du$$
with $R$ bounded by $x=0,y=0,x+y=1$
After changing of variable, I get $$\int_0^{1/\surd2}\int_{-u}^u2v^2e^{-2u^2}dv\,du=\frac{4}{3}\int_0^{1/\surd2}u^3e^{-2u^2}\,du$$
I cannot solve the equation after that part. Help me check which part I made mistake. Thank you.
Let $t=-2u^2$. Then, $u\,du = -\frac{1}{4}dt$ so that \begin{align} \frac{4}{3}\int_0^{1/\surd2}u^3e^{-2u^2}\,du &= \frac{4}{3}\int_0^{1/\surd2}u^2e^{-2u^2}\,u\,du \\ &= \frac{4}{3} \int_0^{-1} \left(-\frac{1}{2}t\right)\,e^t \left(-\frac{1}{4}\right)dt \\ &= \frac{1}{6}\int_0^{-1} t\,e^t \, dt. \end{align} This last integral can be done by parts.
Hint:Use $w = u^2$. Then $dw = 2u\ du$ so
$$\frac{4}{3 \dot\ 2}\int_{0}^{\frac{1}{2}} w e^{-2w} dw $$
Now use integration by parts. I hope you can take it from here.
