Yes, this is true provided that "conjugated" means "conjugate in $GL(2, {\mathbb Z})$". I will leave it to you to construct examples where "conjugate in $SL(2, {\mathbb Z})$" is not enough.
Here is a proof, which is a heavy edit of the original argument that was missing several special cases.
I will use the notation $K={\mathbb Z}^2$ for the normal subgroup in the semidirect product $G_A={\mathbb Z}^2\rtimes _A {\mathbb Z}$ and the notation $Q$ for the quotient $G_A/K$. Note that
$G_A$ is abelian if and only if $A=I$.
$G_A$ is nonabelian but contains an abelian subgroup of index $2$ iff $A=-I$.
(Recall that $A\in SL(2, {\mathbb Z})$.)
Therefore, from now on, I will assume that $A\ne \pm I, B\ne \pm I$.
Under the assumption $A\ne \pm I$, the following dichotomy holds:
(i) Either $A$ fixes a primitive integer vector $v\in {\mathbb Z}^2$; equivalently, $G_A$ contains a unique maximal
normal infinite cyclic subgroup, generated by $v$, or
(ii) $A$ has no eigenvalues $\pm 1$, equivalently, $G_A$ contains a unique maximal
normal rank 2 abelian subgroup, namely, the subgroup $K$.
Equivalence in (i) is clear; to prove equivalence in (ii), note that
any abelian subgroup $H$ strictly containing $K$ will project to an infinite cyclic subgroup of the quotient group $Q:=G_A/{\mathbb Z}^2\cong {\mathbb Z}$, which will imply that $H$ is abelian of rank 3. To show uniqueness, suppose that $H< G_A$ is an abelian subgroup of rank 2. Project $H$ to the quotient group $Q$. If the projection is trivial then $H$ is contained in our subgroup $K$. If the projection is nontrivial, then it is infinite cyclic. Since $H$ has rank 2, $H\cap K$ is also infinite cyclic. Since $H$ is normal in $G_A$,
$H\cap K$ is generated by a primitive integer vector, an eigenvector of $A$.
Consider Case (i). Then either $Av=v$ or $Av=-v$. Which of these two cases occurs depend only on the isomorphism class of the group $G_A$, since $Av=v$ is equivalent to the assumption that $G_A$ has nontrivial center.
(ia) If $Av= v$, then $A$ is conjugate in $GL(2, {\mathbb Z})$ to the matrix
$$
\left[\begin{array}{cc}
1 & n\\
0 & 1\end{array}\right]
$$
In this case, the center $Z(G_A)$ of $G_A$ is infinite cyclic generated by $v\in {\mathbb Z}^2$
and $G_A/Z(G_A)\cong {\mathbb Z}^2$. For any choice of a basis $\{v, w\}$ of ${\mathbb Z}^2$, a generator
$q$ of $Q$ and its lift $u$ to $G_A$, we have
$$
[u, w]= v^{\pm n}.
$$
Therefore, $|n|$ is uniquely determined by the group $G_A$ and, so is the $GL(2, {\mathbb Z})$-conjugacy class of the matrix $A$.
(ib) If $Av=-v$ then $A$ is conjugate in $GL(2, {\mathbb Z})$ to the matrix
$$
\left[\begin{array}{cc}
-1 & n\\
0 & -1\end{array}\right]
$$
Consider then the subgroup $G_{A^2} < G_A$: It is a characteristic subgroup of $G_A$ since it equals the kernel of the action of $G_A$ on $<v>$ (given by conjugation). The subgroup $G_{A^2}$ has nontrivial center and we recover
$|2n|$ from $G_{A^2}$ as in (ia). Thus, again, the group $G_A$ determines the number $|n|$ and, hence, the conjugacy class of $A$ in $GL(2, {\mathbb Z})$.
Case (ii). Thus, $A$ (and $B$) does not have primitive integer eigenvectors in ${\mathbb Z}^2$. Then the subgroup $K$ is the unique maximal normal abelian subgroup of rank $2$ in $G_A$. In order to prove maximality, note that any strictly larger abelian subgroup $H$ will project to an infinite cyclic subgroup of the quotient group $Q:=G_A/{\mathbb Z}^2\cong {\mathbb Z}$, which will imply that $H$ is abelian of rank 3. To show uniqueness, suppose that $H< G_A$ is a normal abelian subgroup of rank 2. Project $H$ to the quotient group
$Q$. If the projection is trivial then $H$ is contained in our subgroup $K$. If the projection is nontrivial, then it is infinite cyclic. Since $H$ has rank 2, $H\cap K$ is also infinite cyclic. Since $H$ is normal in $G_A$,
$H\cap K$ is generated by a primitive integer vector, an eigenvector of $A$, a contradiction.
Thus, $K$ is a characteristic subgroup of $G_A$. Hence, any isomorphism $\phi: G_A\to G_B$ will preserve the semidirect product decomposition. The isomorphism $\phi$ also projects to an automorphism of quotient cyclic groups $\bar{\phi}: {\mathbb Z}\to {\mathbb Z}$. There will be two cases to consider: (i) $\overline\phi: 1\mapsto 1$ and (ii) $\overline\phi: 1\mapsto -1$.
I will identify matrices $A$ and $B$ in your question with corresponding automorphisms of $K$. Each isomorphism $\phi: G_A\to G_B$ will restrict to an equivariant (with respect to the action of $Q$ on $K$) isomorphism $\psi: {\mathbb Z}^2\to {\mathbb Z}^2$ such that (due to equivariance) either
(i) $\psi \circ A= B\circ \phi$, or
(ii) $\psi \circ A= B^{-1} \circ \phi$
(depending on whether we are in Case (i) or Case (ii) above).
Now, use the fact that $Aut({\mathbb Z}^2)= GL(2, {\mathbb Z})$ which allows us to identify the automorphism $\psi$ with a matrix $C\in GL(2, {\mathbb Z})$. Then the above equations translate into:
(i) $CA C^{-1}= B$ or (ii) $C A C^{-1}= B^{-1}$.
qed
