You need to find the factors of the term with the power of 2 and the last term (which can also be in the power of 2)
Let's take this one:
$4a^2 - 16ab - 9b^2$
We know that the factors of the coefficient of the term in $a^2$ are 1, 2 and 4 (we can take the all positives here and ignore the negative factors to make it easier) and for the coefficient of the term in $b^2$, they are 1, 3, 9, -1, -3 and -9.
My method (there are several) is to put the brackets and the variables:
$$(a\ \ \ b)(a\ \ \ b)$$
I try first with 1 and 4 for $a^2$:
$$(a\ \ \ b)(4a\ \ \ b)$$
1 2 3 4
Now, the only possible terms in $ab$ will be $4ab$ (#2 times #3) and $ab$ (#1 times #4), but we need to get $-16ab$ (see original expression), so that we need $-20ab$ to add to the current $4ab$. How can we get $-20$ from the factors 1, 3, 9, -1, -3 and -9, and $4ab$? Doesn't seem possible. So maybe putting #2 as $3b$, meaning #4 has to be $3b$ as well...
$$(a\ \ \ 3b)(4a\ \ \ 3b)$$
But this gives $12ab+3ab$, or if we take the negative, $-12ab-3ab$ not very useful... so probably 1 and 4 are not good. Let's try 2 and 2.
$$(2a\ \ \ b)(2a\ \ \ b)$$
1 2 3 4
Now, the only possible terms in $ab$ will be $2ab$ (#2 times #3) and $2ab$ (#1 times #4), but again we need to get $-16ab$, so that we need $-20ab$ to add to the current $4ab$. How can we get $-20$ from the factors 1, 3, 9, -1, -3 and -9, and $2ab$, $2ab$? Well, here, you can notice we can have $-9\times2ab + 1\times 2ab = -16ab$. So that:
$$(2a-9b)(2a+b)$$
If we expand it, we get: $4a^2 - 18ab + 2ab -9b^2 = 4a^2 - 16ab - 9b^2$
It takes time and a lot of practice to make all the process faster (especially the part where you have to see what numbers go where) and more natural, but if you keep at it, it will remain in your head for a long time!