I have done the 1st part of question and the answer I got is $\frac{5e^4 - 1}{32} $which I verified from the calculator too. But I am confused how to approach to the deducing part using previous result(since it is stated HENCE ). Any help is greatly appreciated.
Substitute $t=x^2$ and you will get:
$$A = \frac 12 \int_1^e t^3(\ln\sqrt{t})^2 dt = \frac 18 \int_1^e t^3(\ln t)^2dt$$
Let $x=u^2$, then $dx = 2u\, du$, and $\ln x = 2\ln u$ and the integral becomes $$ \frac{5e^4-1}{32} = \int_{u=1}^\sqrt{e}8u^3 4(\ln u)^2 2u\,du = 64 \int_1^\sqrt{e} u^7 (\ln u)^2\,du $$ which tels you the integral you were looking to use.
$$\int _{ 1 }^{ e }{ { x }^{ 3 }{ \left( \ln { x } \right) }^{ 2 } } dx=\frac { 1 }{ 4 } \int _{ 1 }^{ e }{ { \left( \ln { x } \right) }^{ 2 } } d\left( { x }^{ 4 } \right) ={ \frac { { x }^{ 4 }{ \left( \ln { x } \right) }^{ 2 } }{ 4 } }_{ 1 }^{ e }-\frac { 2 }{ 4 } \int _{ 1 }^{ e }{ { x }^{ 3 } } \ln { x } dx=\\ =\frac { { e }^{ 4 } }{ 4 } -\frac { 1 }{ 8 } \left( \int _{ 1 }^{ e }{ \ln { x } } d\left( { x }^{ 4 } \right) \right) =\frac { { e }^{ 4 } }{ 4 } -\frac { 1 }{ 8 } \left( { { x }^{ 4 }\ln { x } }_{ 1 }^{ e }-\int _{ 1 }^{ e }{ { x }^{ 3 }dx } \right) =\\ =\frac { { e }^{ 4 } }{ 4 } -\frac { 1 }{ 8 } \left( { e }^{ 4 }-{ \frac { { x }^{ 4 } }{ 4 } }_{ 1 }^{ e } \right) =\frac { { e }^{ 4 } }{ 4 } -\frac { 1 }{ 8 } \left( { e }^{ 4 }-\frac { { e }^{ 4 } }{ 4 } +\frac { 1 }{ 4 } \right) =\frac { 8{ e }^{ 4 }-4{ e }^{ 4 }+{ e }^{ 4 }-1 }{ 32 } =\frac { 5{ e }^{ 4 }-1 }{ 32 } $$
