I am assuming $b>0$. Let's first examine the smallest eigenvalue of $\Delta$ for $u=0$ on boundary: if
$$
-\Delta u = \lambda u
$$
Multiply both side by $u$, integrate:
$$
\int_D -u\Delta u \,dx= \int_D \lambda u^2\,dx \tag{1}
$$
Green's formula is using divergence theorem
$$
\int_D \nabla \cdot \boldsymbol{F} \,dx = \int_{\partial D} \boldsymbol{F}\cdot\boldsymbol{n} \,dS\tag{$\dagger$}
$$
on $\boldsymbol{F} = \psi \nabla \varphi$, thus $\nabla \cdot \boldsymbol{F} = \psi \Delta \varphi + \nabla \varphi \cdot \nabla \psi$ and $(\dagger)$ becomes:
$$\int_D \psi \Delta \varphi \,dx = -\int \nabla \varphi \cdot \nabla \psi\, dx +\int_{\partial D} \psi ( \nabla \varphi \cdot \boldsymbol{n} )\, dS $$
Plugging this formula back to (1), and using $u=0$ on boundary:
$$
\int_D \nabla u\cdot\nabla u \,dx - \int_{\partial D} u ( \nabla u \cdot \boldsymbol{n} )\, dS= \int_D |\nabla u|^2 \,dx= \int_D \lambda u^2\,dx
$$
if $a<\lambda_1$, the smallest possible eigenvalue, this implies for ANY permissible u in some space (in this case $u$ solves the original PDE)
$$
\int_D |\nabla u|^2 \,dx \geq \int_D \lambda_1 u^2\,dx > \int_D a u^2\,dx\tag{2}
$$
Now multiply both sides of
$$
\Delta u+au-bu^2 = 0
$$
by $u$ and integrate over $D$:
$$
\int_D(u\Delta u + au^2 -bu^3) dx = 0
$$
again by Green's identity
$$
\int_D(-\nabla u\cdot \nabla u + au^2 -bu^3) dx + \int_{\partial D} u ( \nabla u \cdot \boldsymbol{n} )\, dS = 0
$$
By $u=0$ on boundary, above integral is:
$$
\int_D(-|\nabla u|^2 + au^2 -bu^3) dx = 0
$$
which is:
$$
\int_D bu^3 \,dx = \int_D(-|\nabla u|^2 + au^2) dx < 0
$$
by (2). Therefore there does not exist a positive $u$ on $D$, otherwise $\displaystyle\int_D bu^3 \,dx >0$.