Indeed, this is true and is known as the fundamental theorem of finitely generated abelian groups..
In particular, we know that every finitely generated abelian group is of the form:
$$\mathbb Z^n \oplus \mathbb Z_p\oplus\cdots \oplus \mathbb Z_q$$
so if your group is infinite and cyclic, then it is abelian, of rank $1$ and torsion-free, meaning that it is indeed the same thing as $\mathbb Z$.
Edit: on the other hand, this now feels like a lot of overkill, despite being the way that I think about it. Here is a proof sketch:
Let $G$ be infinite cyclic $\langle a^n \mid n \in \mathbb Z \rangle $. Suppose further that $\phi: \mathbb Z \to G$ is the map $n \mapsto a^n$.
You should check that this is indeed a homomorphism. Surjectivity is clear. Injectivity follows from the fact that if $a^n=a^m$ for $ n >m$, then $a^n (a^m)^{-1}=e \implies a^{n-m}=e$ while the order of $a$ was supposed to be infinite.
Hence, this is an isomorphism.
Edit 2: Let me try to address your question in the comments. Let $\pi_1(S^1)$ be the group consisting of loop classes, equipped with the usual multiplication. We already have that there exists an isomorphism
$$\rho: \pi_1(S^1) \to (\mathbb Z,+)$$
and we have further that
$$\phi:\mathbb Z \to G$$
is an isomorphism when $G$ is infinite cyclic. By the composition of isomorphisms,
$$\phi \circ \rho:\pi_1(S^1) \to G$$
is an isomorphism as well. Hence, we can conclude that it is infinite cyclic.
This seems like a more general problem, isomorphism is a transitive property, if $A \cong B \cong C$, then $A \cong C$ as well.
Edit 3: We claim that $(\mathbb Z,+)$ is an infinite cyclic group. To see this, consider $1 \in \mathbb Z$. Clearly, every element $n \in \mathbb N$ can be written as $\underbrace{1+1+\dots+1}_{n \, \mathrm{times}}$ and each inverse can be given by $-1$, which is the additive inverse of $1 \in \mathbb Z$. In other words, $1$ generates the group and has infinite order. Perhaps the notation is confusing, in additive notation:
A group $(G,+)$ is said to be infinite cyclic if
$$G=\langle n \cdot a \mid n \in \mathbb Z\rangle.$$
Hence if you already know that $\pi_1(S^1) \cong \mathbb Z$, then it must be infinite cyclic, since $\mathbb Z$ already was.