First, show that it suffices to prove the result for $p$-primary components, for each prime $p|N$. That is, it suffices to show that under your conditions, $A[p^n] \cong (\mathbb{Z}/p^n\mathbb{Z})^r$ for every $p|N$, where $p^n||N$.
Then, look at $p$-primary components and see what it is possible, and what is not. For example, suppose $\# A = (p^2)^2$ and $\#A[p]=p^2$, and $\#A[p^2]=p^4$. Since $A$ is abelian and has size $p^4$, there are only a few possibilities:
$$\mathbb{Z}/p^4\mathbb{Z},\ \text{ or }\ \mathbb{Z}/p^3\mathbb{Z} \times \mathbb{Z}/p\mathbb{Z},\ \text{ or } \ \mathbb{Z}/p^2\mathbb{Z}\times \mathbb{Z}/p^2\mathbb{Z}.$$
What possibility satisfies the hypotheses and why? That should give you the clue about how to finish the proof. (Hint: count elements of order $p$ in each case.)