$
\newcommand{\i}{\hat{\mathbf{i}}}
\newcommand{\j}{\hat{\mathbf{j}}}
\newcommand{\k}{\hat{\mathbf{k}}}
\renewcommand{\p}{\partial\,}
\renewcommand{\b}{\begin}
\renewcommand{\e}{\end}
\renewcommand{\R}{\mathbb R}
$
1. Determine if $\boldsymbol F$ is conservative
HINT: Recall for any scalar vector field $f$ is the curl of its gradient is zero:
$ \nabla \times \big( \nabla f \big) = 0 $.
Since $ F$ is conservative iff $\ \exists\; f : \mathbb R^3 \to \mathbb R$ such that $F = \nabla f$, we can easily check if such $f$ exists by taking curl of $F$ and comparing it to zero
If you are not convinced in the accuracy of the last statement, I propose to address something like this website.
Indeed,
- $\Rightarrow $ If $F$ is conservative, then $\exists$ a scalar field $f $ s.t. $F = \nabla f$.
Taking curl of both sides of the last equality, we get
$ \nabla \times F = \nabla \times \big(\nabla f\big) = 0$.
Thus we established that
$$
\boxed{ \ F \ \text{ is conservative } \implies \nabla \times F = 0 \ }
$$
- $\Leftarrow$ If $\nabla \times F = 0$, then $F$ is conservative, because a vector field is conservative if and only if it is irrotational, which equivalent to $F$ having zero curl in 3D space.
Therefore we can write
$$ \boxed{ \ \nabla \times F = 0 \implies \exists \, f \ \text{ s.t. } \ F = \nabla f \iff F \ \text{ is conservative } \ }$$
1.1 Solution
Now, let us use these facts in order to determine value of parameter $a$ for which your vector field
$$
F =
\begin{bmatrix}
F_x \\ F_y \\ F_z
\end{bmatrix}
=
\begin{bmatrix}
yz-2xy^2 \\ axz-2x^2y+z \\ xy+y
\end{bmatrix}
$$
is conservative.
In other words, we need to find values of $a$ such that the curl of $F$ will be zero.
Denote $\i, \j, \k$ orthonormal basis vectors of $\mathbb R^3$. Then the curl of $F$ can be computed as
$$
\nabla \times F =
\begin{bmatrix}
\i & \j & \k \\
\dfrac{\partial}{\partial x} & \dfrac{\partial}{\partial y} & \dfrac{\partial}{\partial z} \\
F_x & F_y & F_z
\end{bmatrix}
=
\left(\dfrac{\partial F_z}{\partial y}-\dfrac{\partial F_y}{\partial z}\right)\i +
\left(\dfrac{\partial F_x}{\partial z}-\dfrac{\partial F_z}{\partial x}\right)\j +
\left(\dfrac{\partial F_y}{\partial x}-\dfrac{\partial F_x}{\partial y}\right)\k.
$$
Let us compute partial derivatives of $F_x, F_y, F_z$ components of $F$:
$$
\begin{aligned}
\dfrac{\partial F_x}{\partial y} & = z - 4 xy, &
\dfrac{\partial F_x}{\partial z} & = y
\\
\dfrac{\partial F_y}{\partial x} & = a z - 4 x y &
\dfrac{\partial F_y}{\partial z} & = a x + 1
\\
\dfrac{\partial F_z}{\partial x} & = y &
\dfrac{\partial F_z}{\partial y} & = x + 1
\end{aligned}
$$
Substituting it into the $\nabla \times F$ formula, we get
$$
\nabla \times F =
\Big(\left(x+1\right) - \left(ax + 1\right) \Big)\i +
\Big( y - y \Big)\j +
\Big(\left( az - 4 xy \right) - \left(z - 4 xy\right) \Big)\k =
\begin{bmatrix}
\left(1-a\right) x \\ 0 \\ \left( a - 1 \right) z
\end{bmatrix}
$$
If $F$ is a conservation field, then $\nabla \times F = 0 $ for any $(x,y,z) \in \mathbb R^3$, so we write
$$
\nabla \times F
= \begin{bmatrix} \left(1-a\right) x \\ 0 \\ \left( a - 1 \right) z \end{bmatrix}
= \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix} \ \forall \, x,y,z \in \mathbb R \implies \boxed{ \ a - 1 = 0 \ }
$$
Therefore we finally conclude that $F$ is conservation field if and only if $ a = 1 $.
2. Find potential function for $\boldsymbol F$
In the section above we established that $F$ is conservative for $a=1$, i.e. that there exists a potential function $f:\Bbb R^3 \to \Bbb R$ such that
$$
\nabla f =
\left[\begin{matrix}
\frac{\p f}{\p x} \\ \frac{\p f}{\p x} \\ \frac{\p f}{\p x}
\end{matrix}\right]
= F\vert_{a=1} =
\left.\left[\begin{smallmatrix}
F_x \\ F_y \\ F_z
\end{smallmatrix}\right]\right|_{a=1}
=
\begin{bmatrix}
yz-2xy^2 \\ xz-2x^2y+z \\ xy+y
\end{bmatrix}
$$
Thus, one can find potential function $f$ by integrating components of $F$ and matching results.
2.1 Solution
First, integrate $F_x = \dfrac{\p f}{\p x} $ with respect to variable $x$.
$$
\begin{aligned}
\frac{\p f}{\p x} &= yz - 2xy^2
&&\implies& f(x,y,z) &= yz - x^2 y^2 + h_(y,z),
\end{aligned}
$$
where $h$ is unknown functions of $y$ and $z$.
Second, differentiate obtained expression w.r.t. $y$ and compare to $F_y$:
$$
\begin{aligned}
\frac{\p f}{\p y} = F_y
& \implies
\frac{\p }{\p y} \Big(yz - x^2 y^2 + h(y,z)\Big)
=z - 2x^2 y + \frac{\p }{\p y} h(y,z)
= axz - 2 x^2y + z
\\ & \implies
\frac{\p }{\p y} h(y,z) = axz \implies h(y,z) = axyz + g(z)
\\ & \implies
f(x,y,z) = yz - x^2 y^2 + axyz + g(z)
\end{aligned}
$$
where $g$ is unknown functions of $z$.
Third, we differentiate obtained expression for $f$ w.r.t. $z$ and match it to $F_z$:
$$
\begin{aligned}
\frac{\p f}{\p z} = F_z
& \implies
\frac{\p }{\p z} \big(yz - x^2 y^2 + xyz + g(z)\big)
= y + xy + \frac{\p }{\p z} g(z)
= xy + y
\\ & \implies
\frac{\p }{\p z} g(z) = 0 \implies g(z) = c \ \ - \operatorname{const}
\\ & \implies
f(x,y,z) = yz - x^2 y^2 + xyz + c
\end{aligned}
$$
where $c\in \R$ is an arbitrary constant.
Thus we have obtained the potential of $F$:
$$
\boxed{ \ \ f = (1-x) yz - x^2 y^2 + c \ \ }
$$
3. Determine if $\boldsymbol F$ is a curl of another vector field
HINT: Recall that the divergence of a curl of any vector field is zero: $\forall \ A:\R^3 \to \R^3 \quad \nabla \cdot \big( \nabla \times A\big) \equiv 0$.
Therefore in order to establish whether $F$ is a curl of another vector field it is sufficient to check whether $\nabla \cdot F = 0 $ for all $ (x,y,z) \in \R^3$.
2.1 Solution
All we need to do is to compute divergence of $F$, and find out if there values of $a$ for which $\nabla \cdot F \equiv 0$.
$$
\nabla\cdot F =
\dfrac{\p F_x}{\p x} + \dfrac{\p F_y}{\p y} + \dfrac{\p F_z}{\p z} =
\dfrac{\p }{\p x} \Big(yz-2xy^2 \Big)
+ \dfrac{\p }{\p y} \Big( axz-2x^2y+z \Big)
+ \dfrac{\p }{\p z} \Big( xy+y \Big) =
-2y^2 - 2 x^2 + 0
\not \equiv 0
$$
Therefore we conclude that $F$ is not a curl of any other vector field.