Let's first solve the problem for polylines with a fixed (possibly high) number of nodes $P_0=A, P_1, \ldots, P_n=B$. The conditions that the polyline has the desired length ans is convex and within the given triangle are closed, so that the set of tuples $(P_1,\ldots, P_n)$ is compact. As the area in question is a continuous function of the $P_i$, we conclude that the minimum is actually attained.
Let $S_n(p)$ be the minimal area obtainable with a polyline of $n$ nodes.
Consider what happens if we move one node $P_i$ (the top node in the following illustration) of a polyline. By the length condition, $P_i$ is restricted to an elliptical arc with foci $P_{i-1}$ and $P_{i+1}$ (red ellipsis).
The ellipsis may degenerate to a line segment, in which case $Pi$ must be on $P_{i-1}P_{i+1}$ and is redundant. Assume this is not the case.
By the convexity condition, $P_i$ is limited in its position by the prolongations of $P_{i-2}P_{i-1}$ and of $P_{i+2}P_{i+1}$ (red line segments). Similarly, in the case of $i=1$ or $i=n-1$ we have the triangle sides $AC$ or $BC$ as red lines.
If moving $P_i$ under these constraints can make the green triangle smaller, we also make the overall enclosed area smaller. For a minimizing polyline, this is not possible. We conclude that in a minimizing polyline, $P_i$ is at one of the two intersections of the red lines with the ellipsis, i.e., either $P_{i-1}$ is on the line segment $P_{i-2}P_i$ or $P_{i+1}$ is on the line segment $P_iP_{i+2}$ (or, in case of $i=1$: $P_1$ is on $AC$; or, in case $i=n-1$: $P_{n-1}$ is on $BC$). Hence, if $1<i<n-1$, one of the points $P_{i-1},P_i,P_{i+1}$ is redundant.
We conclude that the optimal result for polylines with $n\gg1$ nodes is the same as the optimum for $n=3$, i.e.,
$$ S_n(p)=S_3(p)\qquad \text{for }n\ge 3.$$
And for the latter, we need only consider the cases where $P_1\in AC$ and $P_2\in BC$.
Hence the optimum over all polylines looks somewhat like this:
Let $\theta=\angle P_2P_1C$. Note that $\angle CP_2P_1=90^\circ -\theta$.
Infinitesimally moving $P_1$ so that $u$ changes by an infinitesimal amount $\mathrm du$, will change $v$ by $-\cos\theta\, \mathrm du$ and the green area by $\frac 12 v \sin\theta\, \mathrm du$. Likewise, changing $w$ by an infinitesimal $\mathrm dw$ will change $v$ by $-\sin\theta\, \mathrm dw$ and the green area by $\frac 12 v \cos\theta\, \mathrm dw$.
In order to keep $p$ constant, we must have $(1-\cos\theta)\mathrm du+(1-\sin\theta)\mathrm d w=0$. This makes the change in area
$$ \begin{align}\mathrm dA &= \frac 12 v \sin\theta\, \mathrm du+\frac 12 v \cos\theta\, \mathrm dw\\
&=\frac v2\left(\sin\theta\,\mathrm du+\cos\theta\,\mathrm dw\right)\\
&=\frac v2\left(\sin\theta\,\mathrm du-\frac{1-\cos\theta}{1-\sin\theta}\cos\theta\,\mathrm du\right)\\
&=\left(\sin\theta(1-\sin\theta)-\cos\theta(1-\cos\theta)\right)\frac{v\,\mathrm du}{2(1-\sin\theta)}\\
&=(\sin\theta-\cos\theta)(1-\sin\theta-\cos\theta)\frac{v\,\mathrm du}{2(1-\sin\theta)}\\\end{align}$$
Note that $1-\sin\theta-\cos\theta<0$ for $0^\circ <\theta<90^\circ$. Therefore, $\frac{\mathrm dA}{\mathrm du}$ is negative if $\sin\theta<\cos\theta$ and positive if $\sin\theta>\cos\theta$. We conclude that the minimum is attained when and only when $\sin\theta=\cos\theta$, i.e., when $P_1P_2\|AB$.
Let $h$ be the height of the trapezoid $ABP_2P_1$. Then its bottom line is $\sqrt 2$ and its top is $v=\sqrt 2-2h$. Also, $u=w=h\sqrt 2$, so that
$$h=\frac{p-\sqrt 2}{2\sqrt 2-2} $$
and ultimately
$$ S_3(p)=h\cdot\frac{v+\sqrt 2}{2}=\frac{-p^2+4p+2-4\sqrt 2}{12-8\sqrt 2}.$$
One verifies that this quadratic is an increasing function of $p\in[\sqrt 2, 2]$.
Finally, let us note that the restriction to polylines poses no problems: Assume an arbitrary curve of length $p$ produces an area $\tilde S <S_3(p)$. Then this curve can be approximated by a circumscribed and slightly longer polyline of length $p'>p$, with an area $\tilde S'$ exceeding that for the curve by an arbitrarily small amount (if only we take a large enough number of nodes). In particular, we still have $S_3(p')\le \tilde S'<S_3(p)$, contradicting the fact that $S_3$ is strictly increasing.
Thus ultimately
$$ {S_{\text{opt}}(p)=\frac{-p^2+4p+2-4\sqrt 2}{12-8\sqrt 2}}$$
and the optimum is attained for the trapezoid.
