As you have done, we first use integration by parts to get
$$\begin{align}
I &= \int \frac{\ln(\cos x+\sqrt{\cos 2x})}{\sin^2 x}dx
\\&= \int \ln(\cos x+\sqrt{\cos 2x})\csc^2 (x)dx
\\&=-\cot (x)\ln(\cos x+\sqrt{\cos 2x})+\int \left(\frac{1}{\cos x+\sqrt{\cos 2x}}\right) \left(-\sin x-\frac{\sin 2x}{\sqrt{\cos 2x}} \right)\cot (x)dx
\end{align}$$
Now we just need to deal with the remaining integral which we will call $J$. First note that $$(\cos x+\sqrt{\cos 2x})(\cos x-\sqrt{\cos 2x})=\cos^2x-\cos2x=\cos^2x-(\cos^2x-\sin^2x)=\sin^2x$$
Hence
$$\begin{align}
J&=\int \left(\frac{1}{\cos x+\sqrt{\cos 2x}}\right) \left(-\sin x-\frac{\sin 2x}{\sqrt{\cos 2x}} \right)\cot (x)dx
\\&=\int\left(\frac{\cos x-\sqrt{\cos 2x}}{\cos x-\sqrt{\cos 2x}}\right)\left(\frac{1}{\cos x+\sqrt{\cos 2x}}\right) \left(-\sin x-\frac{\sin 2x}{\sqrt{\cos 2x}} \right)\cot (x) dx
\\&=\int\left(\frac{\cos x-\sqrt{\cos 2x}}{\sin^2x}\right) \left(-\sin x-\frac{2\sin x\cos x}{\sqrt{\cos 2x}} \right)\cot (x) dx
\\&=\int\left(\frac{-\sin x\cos x+\sin x\sqrt{\cos 2x}-\frac{2\sin x\cos^2 x}{\sqrt{\cos2x}}+2\sin x\cos x}{\sin^2x}\right)\cot (x) dx
\\&=\int\frac{-\cos^2 x+\cos x\sqrt{\cos 2x}-\frac{2\cos^3 x}{\sqrt{\cos2x}}+2\cos^2 x}{\sin^2x}dx
\\&=\int\left(-\frac{\sin^2x-1}{\sin^2x}+\frac{\sin^2x-1}{\sin^2x}\right)+\frac{\cos^2x +\cos x\sqrt{\cos 2x}-\frac{2\cos^3 x}{\sqrt{\cos2x}}}{\sin^2x}dx
\\&=-x-\cot x+\int\frac{\cos x\sqrt{\cos 2x}-\frac{2\cos^3 x}{\sqrt{\cos2x}}}{\sin^2x}dx
\\&=-x-\cot x+\int\frac{\cos x\cos 2x-2\cos^3 x}{\sin^2x\sqrt{\cos2x}}dx
\\&=-x-\cot x+\int\frac{\cos x(1-2\sin^2x)-2\cos^3 x}{\sin^2x\sqrt{\cos2x}}dx
\\&=-x-\cot x+\int\frac{-2\cos x\sin^2x+\cos x(1-2\cos^2 x)}{\sin^2x\sqrt{\cos2x}}dx
\\&=-x-\cot x+\int\frac{-\sin2x\sin x-\cos x\cos2x}{\sin^2x\sqrt{\cos2x}}dx
\\&=-x-\cot x+\int\frac{\frac{-\sin2x}{\sqrt{\cos2x}}\sin x-\cos x\sqrt{\cos2x}}{\sin^2x}dx
\\&=-x-\cot x+\frac{\sqrt{\cos 2x}}{\sin x}+C
\end{align}$$
Hence
$$\begin{align}I&=-\cot (x)\ln(\cos x+\sqrt{\cos 2x})-x-\cot x+\frac{\sqrt{\cos 2x}}{\sin x}+C
\\&=\frac{\sqrt{\cos 2x}}{\sin x}-x-\cot (x)(1+\ln(\cos x+\sqrt{\cos 2x})+C
\end{align}$$