The group $G$ is abelian, let us show this.
I will denote by a bar the inverse of an element, so $\bar s$ is $s^{-1}$ to get more compact formulas.
Proposition: Let $G$ be a group with no $2$-torsion elements with the property $abab=baba$ for all
$a,b\in G$. Then $G$ is abelian.
Solution:
Observe first as in the answer of Boris Novikov in (almost) same question
that every element of the shape $x^2$ commutes with any other element of the group,
by using the given relation $(ab)^2 = (ba)^2$ for $b=\bar a x$:
$$
x^2 = (a\cdot \bar ax)^2 = (\bar ax a)^2=\bar a x^2a\ ,
$$
i.e. $ax^2 = x^2 a$.
Let now $s,t$ be two elements in $G$. (We show $st=ts$.)
Let $a$ be the element
$a=\bar s^2\;\color{blue}{(st)^2}\;\bar t^2=\bar s^2\bar t^2\;\color{blue}{(st)^2}$. Then $a^2=1$:
$$
\begin{aligned}
a^2
&=(\bar s^2\bar t^2\;\color{blue}{(st)^2})^2\\
&=\bar s^2\bar t^2\;\color{blue}{(st)^2} \; \bar s^2\bar t^2\;\color{blue}{(st)^2} \\
&=\bar s^4\bar t^4\;\color{blue}{stst\ stst} \\
&=\bar s^4\bar t^4\;\color{blue}{stst\cdot tsts} \\
&=\bar s^4\bar t^4\;\color{blue}{sts\cdot t^2\cdot sts} \\
&=\bar s^4\bar t^4\cdot t^2\;\color{blue}{sts\cdot sts} \\
&=\bar s^4\bar t^4\cdot t^2\;\color{blue}{st\cdot s^2\cdot ts} \\
&=\bar s^4\bar t^4\cdot t^2\cdot s^2\;\color{blue}{st\cdot ts} \\
&=\bar s^4\bar t^4\cdot t^2\cdot s^2\;\color{blue}{s\cdot t^2\cdot s} \\
&=\bar s^4\bar t^4\cdot t^2\cdot s^2\cdot t^2\;\color{blue}{s\cdot s} \\
&=\bar s^4\bar t^4\cdot t^2\cdot s^2\cdot t^2\cdot s^2 \\
&=1\ .
\end{aligned}
$$
So $a$ has finite order, one or two.
Using the asumption on $G$, this implies $a=1$.
So
$$
\begin{aligned}
st
&= s\cdot1\cdot t \\
&=sat\\
&=s\;\bar s^2\; \color{blue}{(st)^2}\;\bar t^2\;t \\
&=s\;\bar s^2\; \color{blue}{stst}\;\bar t^2 \;t \\
&=s\;\bar s^2\; \color{blue}{s\cdot ts\cdot t}\;\bar t^2 \;t \\
&=1\cdot \color{blue}{ts}\cdot 1 \\
&=ts\ .
\end{aligned}
$$
So $G$ is an abelian group.
$\square$
Postlude:
I will type some lines explaining how to find this solution.
The reader in hurry may please skip the long story. But for purely didactical resons the following
may be interesting.
So how to find the above solution?
Fix two elements $s,t\in G$. By restricting to the subgroup generated by $s,t$ inside $G$
we may and do assume that $s,t$ are generating $G$.
Consider now some element of the group $G$.
It can be written as a noncommutative word in the variables $s$, $\bar s$; $t$, $\bar t$. We collect the powers so each element can be written in the shape:
$$
s^nt^ms^kt^l\ldots s^rt^q\ ,$$
with $m,n,k,l,\ldots,r,q\in\Bbb Z$.
(Some of the powers may be zero, so the word may in fact "start with $t$" and/or "end in $s$".)
In each such writing, we split the integer powers of $s,t$ so that an even power $x^2$ occurs, then we move this even power to the beginning of the word.
Below, $w$ (possibly decorated by indices, prime, etc.) will be such a central word generated by $s^2$ and $t^2$, tipically $w=s^{2m}t^{2n}=t^{2n}s^{2m}$ with $m,n\in \Bbb Z$.
Then each element of $G$ is of one of the shapes, written here for psychological reasons so explicitly...
$$
\begin{aligned}
&w\ ,\qquad & \text{ or }\qquad &\\
\text{ or }\qquad&w\cdot s\ ,\qquad & \text{ or }\qquad &w\cdot t\ ,\\
\text{ or }\qquad&w\cdot st\ ,\qquad & \text{ or }\qquad &w\cdot ts\ ,\\
\text{ or }\qquad&w\cdot sts\ ,\qquad & \text{ or }\qquad &w\cdot tst\ ,\\
\text{ or }\qquad&w\cdot stst\ ,\qquad & \text{ or }\qquad &w\cdot tsts\ ,\qquad\text{ (equal elements)}
\end{aligned}
$$
And please note that a further of the shape $w\cdot ststs\ldots$ or $w\cdot tstst\ldots$ is already in one of the above shapes, since for instance $w\cdot ststs=w\cdot (st)^2s=w\cdot (ts)^2s=w\cdot tst\; s^2=(w\; s^2)\cdot tst=w'\; tst$.
With $a,b$ running independently in the above list, all relations $(ab)^2=(ba)^2$ are thus satisfied, if(and only i)f they are for $a,b$ running in the finite list $\mathcal L$
of words obtained from the above by "forgetting about squares" - here especially only $s^2$, $t^2$, and $(st)^2=(ts)^2$ (since they are central, and the relation is in some sense "homogeneuos"):
$$
\mathcal L=\{\ s\ ,\ t\ ;\ st\ ,\ ts\ ;\ sts\ ,\ tst
\ \}\ .
$$
So the solver may now want to consider the following group $H$ - working in it for both purposes in the same time - either obtain a counterexample, or obtain a proof.
$H$ is generated by two "symbols", $S,T$, and has a presentation with relations of the following shape:
- (1) $STST = TSTS$, i.e. $(ST)^2=(TS)^2$,
- (2) $S$ commutes with each element in the list $S^2$, $T^2$, $(ST)^2$,
- (3) $T$ commutes with each element in the list $S^2$, $T^2$, $(ST)^2$,
- (4) all further relations of the shape $(ab)^2=(ba)^2$ for $a,b$ among $S$, $T$, $ST$, $TS$, $STS$, $TST$.
Note that (1) is a special case of (4). Is there any "new" relation provided by (4)?
Also note that in the presence of (1) it is enough to assume only $[S,T^2]=1=[T,S^2]$ instead of (2) and (3), since for instance from here $S$ commutes with $S^2$ (of course), and with $(ST)^2$, explicitly $S\cdot (ST)^2=S\cdot(TS)^2=STSTS=(ST)^2\cdot S$.
So let us work in $H$ for a while.
(Because there is a map $H\to G$, $S\to s$, $T\to t$, so results from $H$ may be "mapped to" results in $G$. Also, if we can furthermore show $(xy)^2=(yx)^2$ for all $x,y\in H$, then $H$ becomes a good candidate for a counterexample, we only need to look furthermore for $2$-torsion elements, if there are any...)
We take a closer look at the relations (4). We let thus $a,b$ run on rows / columns in the table:
$$
\begin{array}{r|c|c|c|c|c|c|}
& S & T & ST & TS & STS & TST\\\hline
S & \blacksquare &1&2&3&4&5\\ \hline
T &1'& \blacksquare &3^*&2^*&5^*&4^*\\ \hline
ST &2'&(3^*)'& \blacksquare &6&7&8\\ \hline
TS &3'&(2^*)'&6'& \blacksquare &8^*&7^*\\ \hline
STS &4'&(5^*)'&7'&(8^*)'& \blacksquare &9 \\ \hline
TST &5'&(4^*)'&8'&(7^*)'&9'& \blacksquare \\ \hline
\end{array}
$$
It remains to analyze the "new" relations only for the cells $1,2,3,\dots,9$ - all other cells correspond to a symmetric case to be analyzed (w.r.t. $a\rightarrow b$ denoted by a prime, and $S\leftrightarrow T$, denoted by a star). Let's see what are the two sides $abab$ and $baba$ obtained in each case.
- $(1)$ - we obtain $STST$ and $TSTS$, they are equal, this is the relation (1) recorded above. Nothing new.
- $(2)$ - we obtain the sides $SSTSST$ and $STSSTS$. Moving the central $SS$ from the middle to the left we obtain equality, nothing new.
- $(3)$ - we obtain the sides $STSSTS$ and $TSSTSS$. Moving the central $SS$ from the middle to the left we obtain equality, nothing new.
- $(4)$ - we obtain the sides $SSTSSSTS$ and $STSSSTSS$. Moving the central $SS$ from the middle to the left we obtain $SSTSSSTS=S^4\;TSTS$ and $STSSSTSS=S^4\;STST$, by (1) these elements are already equal, nothing new.
- $(5)$ - we obtain the sides $STSTSTST=(ST)^4$ and $TSTSTSTS=(TS)^4$, by (1) these elements are already equal, nothing new.
- $(6)$ - we obtain the sides $STTSSTTS$ and $TSSTTSST$, we move $SS$ and $TT$ to the left end, get $STTSSTTS=S^2\; STTTTS= S^2T^4\; SS=S^4T^4=T^4S^4$ and similarly $TSSTTSST=S^4T^4$, nothing new.
- $(7)$ - we obtain the sides $STSTS\; STSTS$ and $STSST\;STSST$, we use (1) or move again $SS$ to the left end, to get $STSTS\; STSTS= STST\; S^2\; TSTS=S^2\;STST\; TSTS=S^2T^2\; STS\; STS=S^4T^2\; ST\; TS=S^4T^4\; S\; S=S^6T^4$, and $STSST\;STSST=S^4\; STT\; STT=S^4T^4\; SS=S^6T^4$, nothing new again.
- $(8)$ - we obtain the sides $STTST\; STTST$ and $TSTST\;TSTST$, which is similar to $(7)$, a sort of $(7^*)$.
- $(9)$ - we obtain the sides $STSTST\; STSTST=(ST)^6$ and $TSTSTS\;TSTSTS=(TS)^6$, we use (1) to see they are equal, nothing new again.
So the group $H$ from above has a simpler presentation:
$$
\tag{$\dagger$}
H=\langle\ S,T\ |\ S^2T=TS^2\ ,\ T^2S=ST^2\ ,\ (ST)^2=(TS)^2\
\rangle\ .
$$
From now on we will work with this form of $H$. Observe that the morphisms
$$
\begin{aligned}
\deg_S\ &:\ H\to\Bbb Z\ , &\deg_S(S) &= 1\ , & \deg_S(T) &= 0\ ,\\
\deg_T\ &:\ H\to\Bbb Z\ , &\deg_S(S) &= 0\ , & \deg_S(T) &= 1\ ,\\
\deg\ &:\ H\to\Bbb Z^2\ , &\deg &= (\deg_S,\deg_T)\ ,
\end{aligned}
$$
are well defined and surjective.
With the already used ideas one can show:
Lemma: For all $a,b\in H$ we have $abab=baba$.
Proof: Write $a=wa'$, where $w$ is generated by the central elements $S^2$, $T^2$, $(ST)^2$,
so that $a'$ is an elemnent of the list $\{1,S,T,ST,TS,STS,TST\}$.
Then $abab=baba$ is equivalent to $a'ba'b=ba'ba'$.
So we may and do assume $a=a'$ is in the list. Similarly, $b$ is also in this list.
If $1$ is among $a,b$ things are trivial. Else we have again to analyze the cases $(1)$ to $(9)$ from above.
$\square$
Lemma: If $a\in H$ has finite order, then $a=1$ or $a=S^{-2}T^{-2}\; STST$.
Proof: Assume $a\in H$ has finite order. Then $\deg(a) = (0,0)$, we try to use this.
We write $a=wa'$, where $w$ is a product of integer powers $S^{2m}$, $T^{2n}$, $(ST)^{2k}$
of the central elements $S^2$, $T^2$, $(ST)^2$.
The power $k$ of the last element, $(ST)^2$, may be taken among $0,1$ only,
since in case of a power $k=\pm2$, $\pm 3$, ... we can rewrite
$$
\begin{aligned}
(ST)^4
&=(STST)(STST)\\
&=(STST)(TSTS)\\
&=(STS)\; T^2\; (STS)\\
&= T^2\; (STS)(STS)\\
&= S^2T^2\; (ST)(TS) \\
&= S^2T^4\; SS\\
&= S^4T^4\ .
\end{aligned}
$$
and replace $k$ by $k$ modulo two. And $a'$ is an element of the list:
$$
1\ ,\
S\ ,\
T\ ,\
ST\ ,\
TS\ ,\
STS\ ,\
TST\ .
$$
From $(0,0)=\deg(a) = \deg(wa')=\deg(w)+\deg(a')$ and $\deg(a')=-\deg(w)\in 2\Bbb Z\times 2\Bbb Z$,
we see that only $a'=1$ is a match in the above list.
This shows that $a=wa'$ is a product of the central elements of the shape
$a = S^{-2k}T^{-2k}(ST)^{2k}$. And $k=0,1$ points to two value from the Lemma.
$\square$
So we are naturally obliged to consider the element:
$$
a := S^{-2}T^{-2}\; STST= S^{-2}T^{-2}\; TSTS\ ,
$$
and consider also $a^2$, the only element which may matter while verifying the conditions imposed in the question.
Well,
$$
\begin{aligned}
a^2
&= (S^{-2}T^{-2}\; \color{blue}{STST})^2\\
&=
S^{-2}T^{-2}\; \color{blue}{STST}\; S^{-2}T^{-2}\; \color{blue}{STST}\\
&=
S^{-2}T^{-2}S^{-2}T^{-2}\; \color{blue}{STST\; STST}\\
&=
S^{-2}T^{-2}S^{-2}T^{-2}\; \color{blue}{STST\; TSTS}\\
&=
S^{-2}T^{-2}S^{-2}T^{-2}\; \color{blue}{STS\; T^2\; STS}\\
&=
S^{-2}T^{-2}S^{-2}\; \color{blue}{ST\; S^2\; TS}\\
&=
S^{-2}T^{-2}\; \color{blue}{S\; T^2\; S}\\
&=
S^{-2}\; \color{blue}{S^2}\\
&=1\ .
\end{aligned}
$$
So $a^2=1$.
At this point we can finally decide if we are searching for a counterexample (based on $H$),
or if we are searching for a proof (based on the above element $a$). The solution was given in the first lines.
One more comment:
It may be interesting to obtain a representation $\varphi$ of $H$. (Since it may also lead to the solution.)
One way to realize a representation is by using the
multiplicative group of the field $F$ of rational quaternions.
Recall that
$F$ is as a vector field over $\Bbb Q$ the sum
$
\Bbb Q\cdot 1\oplus
\Bbb Q\cdot i\oplus
\Bbb Q\cdot j\oplus
\Bbb Q\cdot k
$, and the multiplication, the algebra structure, is given by setting as usual $i^2=j^2=k^2=-1$, $ij=k=-ji$, $jk=i=-kj$, $ki=j=-ik$.
Then set for instance $\varphi(S)=s:=2i$, and $\varphi(T)=t:=3j$.
The norm provides a map to $\Bbb Q^\times$, and the image is $\cong\Bbb Z^2$.
There is also a map into the group $W=\{\pm 1,\pm i,\pm j,\pm k\}$, $s\to i$, $t\to j$, and then into $W/\pm 1$.
This quotient is commutative. So we can write group maps $H\to\Bbb Z^2\times W\to\Bbb Z^2\times W/\pm 1$.
Is there any torsion element in the image of $\varphi$? If yes, then it has norm one.
So it lands via $H\to\Bbb Z^2$ in $(0,0)$, this implies that it lands via
$H\to \Bbb Z^2\times W/\pm 1$ in $(0,0,\pm 1)$. So only $-1$ is a possible torsion element!
Of course it is easy to realize it. The corresponding preimage in $H$ leads to the solution.
