Find the probability that if three cards are drawn from a well-shuffled deck that none of the cards are hearts.
You asked why we could not use the complement rule here. We can. However, the complement of the set of outcomes in which none of the cards are hearts is not the set of outcomes in which all the cards are hearts. For instance, the hand $5\color{red}{\heartsuit}7\color{red}{\diamondsuit}J\clubsuit$ contains a heart, but not all of the cards in the hand are hearts. The complement of the set of outcomes in which none of the outcomes are hearts is the set of outcomes in which at least one of the cards is a heart.
Since the order of selection does not matter, this problem is best handled with combinations. The number of ways of selecting a subset of $k$ elements from a set with $n$ elements is
$$\binom{n}{k} = \frac{n!}{k!(n - k)!}$$
Since $13$ of the $52$ cards in the deck are hearts, $52 - 13 = 39$ are not. Hence, a favorable hand consists of drawing $3$ of the $39$ cards that are not hearts when selecting $3$ of the $52$ cards in the deck. Hence, the desired probability is
$$\Pr(\text{none of the cards is a heart}) = \frac{\dbinom{39}{3}}{\dbinom{52}{3}} = \frac{\dfrac{39!}{3!36!}}{\dfrac{52!}{3!49!}} = \frac{39!}{3!36!} \cdot \frac{3!49!}{52!} = \frac{39 \cdot 38 \cdot 37}{52 \cdot 51 \cdot 50}$$
We can also compute the probability by subtracting the probability of the complementary event from $1$. As stated above, the complementary event is the event that at least one of the selected cards is a heart. The number of ways of selecting $k$ of the $13$ hearts and $3 - k$ of the other $39$ cards in the deck is
$$\binom{13}{k}\binom{39}{3 - k}$$
Hence, the probability that at least one of the three selected cards is a heart is
$$\Pr(\text{at least one the cards is a heart}) = \frac{\dbinom{13}{1}\dbinom{39}{2} + \dbinom{13}{2}\dbinom{39}{1} + \dbinom{13}{3}\dbinom{39}{0}}{\dbinom{52}{3}}$$
Therefore, the probability that none of the selected cards is a heart is
$$\Pr(\text{none of the cards is a heart} = 1 - \frac{\dbinom{13}{1}\dbinom{39}{2} + \dbinom{13}{2}\dbinom{39}{1} + \dbinom{13}{3}\dbinom{39}{0}}{\dbinom{52}{3}}$$
Find the probability that the cards are not all hearts.
This probability is found by subtracting the probability that all of the cards are hearts from $1$, which is
$$\Pr(\text{not all of the cards are hearts}) = 1 - \frac{\dbinom{13}{3}}{\dbinom{52}{3}}$$
The complementary event is the probability that at least one of the cards is a heart, which we computed above.
You asked why this probability is not
$$\frac{13}{52} \cdot \frac{39}{51} \cdot \frac{38}{50} + \frac{13}{52} \cdot \frac{12}{51} \cdot \frac{39}{50} + \frac{13}{52} \cdot \frac{12}{51} \cdot \frac{11}{50}$$
You corrected calculated the probability of selecting three hearts in three draws. However, when you calculated the probabilities of selecting exactly one heart in three draws or exactly two hearts in three draws, you did not account for the order in which the cards are drawn.
For instance, when you calculated the probability of selecting one heart in three draws, you calculated the probability of first selecting a heart and then selecting two non-hearts. However, the heart also could be drawn on the second or third draw, so the probability of selecting exactly one heart in three draws is actually
$$\frac{\dbinom{13}{1}\dbinom{39}{2}}{\dbinom{52}{3}} = \frac{13}{52} \cdot \frac{39}{51} \cdot \frac{38}{50} + \frac{39}{52} \cdot \frac{13}{51} \cdot \frac{38}{50} + \frac{39}{52} \cdot \frac{38}{51} \cdot \frac{13}{50}$$
Similarly, the probability of selecting exactly two hearts in three draws is
$$\frac{\dbinom{13}{2}\dbinom{39}{1}}{\dbinom{52}{3}} = \frac{13}{52} \cdot \frac{12}{51} \cdot \frac{39}{50} + \frac{13}{52} \cdot \frac{39}{51} \cdot \frac{12}{50} + \frac{39}{52} \cdot \frac{13}{51} \cdot \frac{12}{50}$$