If $A$ is the set containing the positive integer factors of $12$ and $B$ is the set containing the positive integer factors of $16$, how many different numbers are in both sets $A$ and $B$?
(Actually the answer says there are three different numbers but no matter how many times I count, I get that there are five. Why is the answer three?)