Your first substitution of $t=\tan{x}$ is correct, but as @metamorphy said, you must adjust the bounds to because it is not one to one with the $\tan{x}$ substitution (and this works because of symmetry with $\sin^2{x}$ from $0$ to ${\pi}$ is twice of $\sin^2{x}$ from $0$ to $\frac{\pi}{2}$.
$\frac{dt}{t^2+1}=dx$ and $\sin^2{x}=\frac{t^2}{t^2+1}$:
$I=2\int_0^{\infty} \frac{\frac{dt}{t^2+1}}{1+\frac{a^2t^2}{t^2+1}}=2\int_0^{\infty} \frac{dt}{\left(a^2+1\right)t^2+1}=\frac{2}{\sqrt{a^2+1}}\arctan{\left(t\sqrt{a^2+1}\right)} \big\rvert_0^{\infty}=\frac{2}{\sqrt{a^2+1}} \cdot \frac{\pi}{2}=\boxed{\frac{\pi}{\sqrt{a^2+1}}}$
Hope this helps.