Whether you actually get a triangle depends on which of the points $(1,1)$ and $(-1,-1)$ lies above the box or below. So to distinguish these cases, let's see when these points lie on the line.
\begin{align*}
(1,1)&:& 1=m(b+1)&\implies m+mb=1 \implies b=\frac1m-1 \\
(-1,-1)&:& -1=m(b-1)&\implies m-mb=1 \implies b=1-\frac1m
\end{align*}
So now you can distinguish four cases:
- $0<m<1,\quad 0<b<\frac1m-1,\quad (1,1)$ lies above and $(-1,-1)$ lies below the line
- $0<m<1,\quad \frac1m-1<b<\infty,\quad (1,1)$ lies below and $(-1,-1)$ lies below the line
- $1<m<\infty,\quad0<b<1-\frac1m,\quad (1,1)$ lies below and $(-1,-1)$ lies above the line
- $1<m<\infty,\quad1-\frac1m<b<\infty,\quad (1,1)$ lies below and $(-1,-1)$ lies below the line
As you can see, there are two of these four cases where the two mentioned corners both lie below the line, so that the line actually cuts away a triangle as opposed to a right trapezoid. Its one corner is $(-1,1)$, the other two can be computed as
\begin{align*}
x&=-1 & y&=m(b-1) & 1-y&=1-m(b-1) \\
y&=1 & x&=\frac1m-b & 1+x&=1+\frac1m-b
\end{align*}
The last column lists the legs of the right triangle cut from your square. Multiply these and divide by two to get the area:
$$A=\frac12\biggl(1-m(b-1)\biggr)\biggl(1+\frac1m-b\biggr)$$
This is almost the formula you mention in your question, except for order (which does not matter), a change of sign and the $\frac12$ in front.