Question:
Let $H$ be a subgroup of $G$. For any $a \in G$, let $aHa^{-1} = \{axa^{-1} : x \in H\}$; $aHa^{-1}$ is called a conjugate of $H$. Prove: For each $a \in G$, $aHa^{-1}$ is a subgroup of $G$.
I know in order to prove something is a subgroup it needs to be nonempty, closed under operation, contains it's identity, and inverse.
I do believe to prove H contains it's inverse it goes as follows:
$aea^{-1} = aa^{-1}e = e$ for any $a \in G$. Therefore $e \in H$.
but after that I am at a loss.
Let $\alpha, \beta\in aHa^{-1}$, then $\exists x,y \in H$ such that $\alpha=axa^{-1}$ and $\beta=aya^{-1}$. Note that $aHa^{-1}\neq \emptyset$ because $e\in H$ (identity oh $H$), then $e=aa^{-1}=aea^{-1}\in aHa^{-1}$.
Then, $$\alpha \beta^{-1}=(axa^{-1})(aya^{-1})^{-1}=(axa^{-1})((a^{-1})^{-1}y^{-1}a^{-1})= $$ $$=(axa^{-1})(ay^{-1}a^{-1})=ax(a^{-1}a)y^{-1}a^{-1} =axy^{-1}a^{-1}=a(xy^{-1})a^{-1}\in aHa^{-1},$$ because $x,y\in H<G\Rightarrow xy^{-1}\in H<G$.
Then, $aHa^{-1}<G$, $\forall a\in H$.
Note: $H<G$ means that $H$ is a subgroup of $G$.
You've shown $aHa^{-1}$ contains the identity $e$, actually. And so it's certainly nonempty. That's a third of the battle.
To show $aHa^{-1}$ is closed under inverses, consider an element $aha^{-1}$. What is its inverse in $G$? Is that also in $aHa^{-1}$?
Suppose you have two elements $aha^{-1}$ and $ah'a^{-1}$ of $aHa^{-1}$. Can you show their product is in $aHa^{-1}$?
One slick way to prove a subset $K$ of a group $G$ is a subgroup is to show:
1) The set $K$ is nonempty.
2) For every $x,y \in K, xy^{-1} \in K$.
(See if you can prove this criterion to be a subgroup!)
You've shown 1), since the identity $e \in K = aHa^{-1}$. Now let $x,y \in aHa^{-1}$. Then $x = aha^{-1}$ and $y = ah'a^{-1}$ for some $h,h' \in H$. But $y^{-1} = ah'^{-1}a^{-1}$ (check if you aren't familiar with this formula). Hence $xy^{-1} = aha^{-1}ah'^{-1}a^{-1} = ahh'^{-1}a^{-1}$. Since $H$ is a subgroup of $G$, $hh'^{-1} \in H$, so $xy^{-1} \in aHa^{-1}$, which completes the proof.
