I attempted to plot the integral in question on Desmos. Although I do not believe that $h(x) = x^2$ makes all that much sense in the context of a probability density function, I was first curious to examine the case when $h(x) = x^2$ and $k = 2$.
\begin{align}
\int_{-\infty}^{\infty} \frac{h(x)}{(h(x) + \lambda|x|^k)^2} dx &= \int_{-\infty}^{\infty} \frac{x^2}{(1 + \lambda)^2|x|^{4}} dx \\
& = \int_{-\infty}^{\infty} \frac{x^2}{(1 + \lambda)^2|x|^{4}} dx \\
& = \frac{2}{(1 + \lambda)^2} \int_{0}^{\infty} \frac{1}{x^{2}} dx
\end{align}
This last integral clearly does not converge as $\lim_{x\to 0} \frac{1}{x} \to \infty$. I would like to believe, furthermore, that such behavior will be avoided when $h(x) < 1 \forall x \in \mathbb{R}$, but I am not certain of it. (I think one may also need to ensure that the PDF has only finite support. The conditions together would probably be sufficient but not necessary for the integral to exist.)
I guess that it would be prudent to consider the most common distribution
$$
f(x) = \frac{1}{\sigma \sqrt{2 \pi}} \exp \left({\frac{-1}{2} \left(\frac{x-\mu}{\sigma}\right)^2}\right)
$$
I am just interested in the shape of the function, so I shall simplify it to
$$
f(x) = \exp \left({\frac{-x^2}{2}}\right)
$$
Then, taking a second order Taylor approximation
$$
f(x) = 1 - \frac{x^2}{2}
$$
we find that it is only really valid in the neighborhood $\left(-\frac{1}{2}, \frac{1}{2}\right)$. (Note I haven't considered the error bound rigorously, I only examine it graphically.)
We can then restrict the support of the integrand to that interval and note that
$$\int_{-\frac{1}{2}}^{\frac{1}{2}} \frac{1}{1 - x^2} dx$$
converges (and can be thought of as a constant multiple of an arctangent like integral along the imaginary axis). For the case of the full distribution, we may find an apparent contradiction if we assume uniform convergence and pass the limit inside of the integral.
\begin{align}
\lim_{\lambda\to 0} \int_{-\infty}^{\infty} \frac{h(x)}{(h(x) + \lambda|x|^k)^2} dx &= \int_{-\infty}^{\infty}\lim_{\lambda\to 0} \frac{e^{-\frac{x}{2}}}{\left(e^{-\frac{x}{2}} + \lambda|x|^2\right)^2} dx \\
&= \int_{-\infty}^{\infty} \frac{1}{e^{-\frac{x}{2}}} dx\\
&= 2 \int_{0}^{\infty} e^{\frac{x}{2}} dx
\end{align}
But, unfortunately, this last integral fails to converge as well, seemingly invalidating my original hypothesis about boundedness implying well posedness. I would guess that my final question is
If $h(x)$ is square integrable and decays to $0$ asymptotically, would not the integrand be approximated as $\frac{1}{\lambda |x|^k}$ for sufficiently large $x>x_0$? Further, I feel that the rate at which $\lambda\to 0$ would be important in evaluating the limit. I wonder if the term proportional to $\frac{1}{\lambda} \frac{1}{x^{k-1}}$ could be problematic.
I think the integral would only make sense (for a general PDF) when the domain is finite. I might be grossly mistaken here, but these were my first thoughts. Did this problem appear in a textbook or did you come across it while researching? If the latter is true, I might wish to suggest MathOverflow where you could pose the problem to a specialist.
Specifically, if you divide top and bottom by h(x)², the denominator is like a continuous approximation of an indicator, so it "feels" like we're integrating 1/h(x) over a neighborhood of the origin, the radius determined by λ. But I'm not sure how to handle it.
– Donkey Fronkey Aug 31 '22 at 02:38