The biggest problem is that there are only two prime factors of each RSA modulus, and that primes are fairly abundant in any given interval (http://en.wikipedia.org/wiki/Prime_number_theorem). Sure, you are only interested in odd primes, and can eliminate many options once you've deduced 3 doesn't factor. But think about this: a 1024-bit RSA number $n$ will have prime factors somewhere between 0 and $\sqrt(n)$. However, by the prime number theorem, this interval will contain approximately $1\times 10^{171}$ primes. Furthermore you don't a priori know which of those numbers are prime! Hence you have to check each number in the interval to see if it is prime, and then check to see if it factors $n$. Hence you wind up with an algorithm that is $\mathcal{O}(\sqrt{n})$, where $n$ is unfeasibly large.
Now, there are ways to speed up both parts of the above process by using number fields (the number field sieve), but even this very sophisticated algorithm is impractically slow for most RSA moduli.