Following @SharkyKesa's hint, observe that if you apply a linear transformation to your triangle (and the three selected points), the ratio between the area of the subtriangle and the main triangle remains invariant. So you can begin by transforming the main triangle to any one that's convenient to work with. I propose one with vertices $A = (0,0), B = (1, 0), C = (0, 1)$, whose area is $T = \frac12$.
Picking vertices for the subtriangle in each edge in turn gives
$$
v_1 = (0, t)\\
v_2 = (u, 0)\\
v_3 = (1-s, s).
$$
where $s,t,u$ are uniform random variables on $[0, 1]$.
Note that $v_1v_2v_3$ is a counterclockwise traversal of the triangle.
Following the discussion here, the area of the triangle $v_1v_2v_3$ is
\begin{align}
S
&= \frac12\left(-ut + us + (1-s)t \right)\\
2S&= u(s-t) + t - st\\
\end{align}
To compute the expected value of $2S$, we integrate over $[0, 1]^3$ to get
\begin{align}
avg(2S)
&= \int_0^1 \int_0^1 \int_0^1 u(s-t) + t - st ~du ~dt ~ds\\
&= \int_0^1 \int_0^1 \left. (\frac{u^2}{2}(s-t) + u(t - st)) \right|_{u=0}^1 ~dt ~ds\\
&= \int_0^1 \int_0^1 \frac{1}{2}(s-t) + (t - st) ~dt ~ds\\
&= \int_0^1 \int_0^1 \frac{1}{2}s + \frac12 t - st ~dt ~ds\\
&= \int_0^1 \left. \frac{1}{2}st + \frac12 \frac{t^2}{2} - s\frac{t^2}{2} \right|_{t=0}^1 ~ds\\
&= \int_0^1 \frac{1}{2}s + \frac14 - \frac{s}{2} ~ds\\
&= \int_0^1 \frac14~ds\\
&= \frac14
\end{align}
so that the average of $S$ is $\frac{1}{8}$, which is $\frac{1}{4}T$, where $T = \frac12$ is the area of the original outer triangle $ABC$.