Mistakes in attempt
$x^{-\frac 12}e^{-\frac{x^2}{k}} \leq e^{-\frac 1k}$ is false, for example as $x$ approaches $0$ the LHS approaches infinity while the RHS stays fixed.
The limit is incorrect on the last line.
To apply the DCT , we want to make sure that we know whether the function $\lim_{k \to \infty} x^{-\frac 12}\cos(x^k)e^{-x^2/k}$ exists or not as an a.e. pointwise limit on $[0,1]$, and what it is.
To do this, we look at the parts.
$x^{-\frac 12}$ doesn't depend on $k$, so it stays as is.
What does $x^k$ look like for large $k$ and $x\in [0,1]$? What is this part going to converge a.e. to?
It would be the function $f(x) = 0$ for $x \neq 1$, and $f(1) =1$. So it's a.e. zero, in other words.
- Hence, what does $\cos(x^k)$ converge a.e. to?
Since the cosine is continuous, $\cos(x^k)$ would converge to $1$ a.e.
- Where does $-\frac{x^2}{k}$ converge to a.e. ?
Obviously it converges to the zero function.
- Hence, where does $e^{-\frac{x^2}{k}}$ converge?
Since the exponential is continuous, close to $1$.
Thus, putting things together, we get by the product rule that $\lim_{k \to \infty} x^{-\frac 12}\cos(x^k)e^{-x^2/k}$ equals:
$x^{-\frac 12} \times 1 \times 1 = x^{-\frac 12}$ a.e.
and thus, providing the DCT held, the answer is:
$\int_0^1 x^{-\frac 12}dx = 2$.
How to think about the DCT? We obviously want to bound the function $|x^{-\frac 12}\cos(x^k)e^{-x^2/k}|$ uniformly in $k$ and get an integrable function of $x$ as the bound.
- What is the obvious bound for $|\cos(x^k)|$?
It's $1$, isn't it?
- What is an upper bound for $-\frac{x^2}{k}$ for any $k$ and $x \in [0,1]$?
Come on, they're all negative quantities, so $0$!
- Thus, what is an upper bound for $e^{-\frac{x^2}{k}}$?
By monotonicity of the exponential, the required upper bound is $e^{0}=1$.
- Leaving the first term as is, what does the eventual bound look like? Is it integrable?
$x^{-\frac 12} \times 1 \times 1 = x^{-\frac 12}$, obviously integrable!
Of course it turns out to be integrable, but it also turns out to be ... yeah, it's in the spoiler.
Here's the integral for $k=25$ computed via Wolfram Alpha, and you can check that it's already quite close to the actual answer we derived.