0

In the literature of sequential Monte Carlo, the probability density function (PDF) of a random variable $X$ is often stated as equal to the expectation of the Dirac delta "function" shifted by $X$:

$$f_X(x)=\mathbf E[\delta(x-X)]$$

Here is a blog post explaining this.

Suppose that $X$ has a PDF. Then, the expectation of any function $g$ of $X$ is

$$\mathbf E[g(X)]=\int_{-\infty}^\infty g(x)f_X(x)dx$$

Let $g(x)=\delta(y-x)$ for some $y\in\mathbf R$. Then,

$$\mathbf E[\delta(y-X)]=\int_{-\infty}^\infty\delta(y-x)f_X(x)dx=f_X(y)$$

Since $y$ is arbitrary, this holds for all $y\in\mathbf R$.

We know that a real-valued function of a random variable is a random variable, so we may speak of its expectation. However, $\delta(x-X)$ is really a function that maps a random variable $X$ to a "function": $X\mapsto x\mapsto\delta(x-X)$. Strictly speaking, $\delta$ is not a even a function. How can we speak of the expectation of something that is not a random variable?

The empirical PDF of a sample $x_1,x_2,\ldots,x_n$ is defined (by "replacing the expectation with the average") as:

$$\frac1n\sum_{i=1}^n\delta(x-x_i)$$

Is this really a PDF? In my opinion, if such "functions" were accepted as "PDFs", then every random variable would have a "PDF".

W. Zhu
  • 1,325
  • Yes, every random variable has a density with respect to its pushforward measure, trivially. – Andrew Aug 05 '23 at 04:10
  • @AndrewZhang Could you explain what you mean by a density and a pushforward measure? – W. Zhu Aug 05 '23 at 04:50
  • An example of a continuous random variable that does not have a PDF is the Cantor random variable according to an answer at https://math.stackexchange.com/questions/3760521/does-every-continuous-random-variable-have-a-pdf. – W. Zhu Aug 05 '23 at 06:07
  • I think that this very bad abuse of notation. As you noted, $\delta_x$ (the Dirac mass at the point $x$) is not a function. Instead, it is a distribution (also called "generalized function" sometimes. It is theoretically possible to define the expectation, but it is not the value of the probability density. Instead, one has to interpret it as $$ \int_{\mathbb R} g(x)f_X(x) dx = \mathbb E[g(X)] = \mathbb E[ \int_{\mathbb R} g(x)\delta_x(X) dx] "=" \int_{\mathbb R} g(x)\mathbb E[ \delta_{x}(X)] dx.$$ In this "weak" sense, the equality holds. – Mushu Nrek Aug 05 '23 at 09:34
  • @W.Zhu $P(X\in B)=\int_B d\mu$ for $\mu(B)=P(X\in B)$ – Andrew Aug 05 '23 at 10:54
  • @MushuNrek How can we define the expectation of $\delta_x(X)$? If it is not the value of the probability density, then are you of the opinion that probability density cannot be expressed as an expectation? – W. Zhu Aug 19 '23 at 02:58
  • @Andrew Please elaborate why you think that every random variable has a density. Based on my understanding, the cumulative distribution function (CDF) is defined as $F_X:\mathbf R\to[0,1],F_X(x)=\mathbf P(X\le x)$, and the probability density function (PDF) is defined as its derivative: $f_X:\mathbf R\to\mathbf R,f_X(x)=\frac{dF_X}{dx}$. Where the CDF is not differentiable, the density is not defined. – W. Zhu Aug 19 '23 at 03:07
  • Read my first comment. When one says density, one needs to specify a reference measure. This can trivially always be taken to be the pushforward – Andrew Aug 19 '23 at 09:50
  • As another example, any discrete random variable by definition has a density with respect to the counting measure – Andrew Aug 19 '23 at 10:16
  • @Andrew Could you please provide your definition of density? – W. Zhu Aug 19 '23 at 11:18
  • (Koralov and Sinai) "In some cases (not always!) there exists a non-negative integrable function $p(t)$ on the real line such that $F(x)=\int_{-\infty}^xp(t)dt$ for all $x$. In this case $p$ is called the probability density of $F$ or simply the density of $F$. If $F=F_{\xi}$ is the distribution function of a random variable $\xi$, then $p=p_{\xi}$ is called the probability density of $\xi$. While the right-hand side of the formula above in general should be understood as the Lebesgue integral . . . , for continuous densities $p(t)$ it happens to be equal to the usual Riemann integral." – W. Zhu Aug 19 '23 at 11:22
  • Density is probabilistic jargon for radon nikodym derivative – Andrew Aug 19 '23 at 11:46
  • Stylistically, one may prefer to use RN derivative over density so as to avoid confusion, since density is reserved for Lebesgue measure in non-measure theoretic settings. Though in general they are used interchangeably, with the understanding that if no reference measure is specified, it is Lebesgue – Andrew Aug 19 '23 at 11:56
  • @W.Zhu It is actually not straight-forward to do so. If you want to have more information on this subject, I recommend you google "Vector Measures". But I really don't think that this is directly useful to you. Instead, this is just a case of abuse of notation. But it remains an interesting subject on its own! – Mushu Nrek Aug 19 '23 at 14:03
  • @MushuNrek I just realized that probability mass functions (PMFs) and cumulative distribution functions (CDFs) can be expressed as expectations. Let $X$ be a discrete random variable. The Kronecker delta function of $x$ at $X$ (which is $1$ if $X=x$ and $0$ otherwise) is Bernoulli with parameter $p=\mathbf P(X=x)$, so its expectation is $\mathbf P(X=x)$. Now, let $X$ be any random variable. The step function of $x$ at $X$ (which is $1$ if $X<x$ and $0$ otherwise) is Bernoulli with parameter $p=\mathbf P(X\le x)$, so its expectation is $\mathbf P(X\le x)$. Only the PDFs are problematic, right? – W. Zhu Aug 23 '23 at 02:51
  • @Andrew When we talk about a random variable, or a real-valued measurable function $X:\Omega\to\mathbf R$, we usually talk about the probability of ${\omega:X(\omega)\le x}$ for some $x\in\mathbf R$. That probability as a function of $x$ is the cumulative distribution function (CDF) of $X$: $F_X:\mathbf R\to[0,1]$. The probability density function (PDF) is the derivative of the CDF: $\frac{dF_X}{dx}$ (if defined). How is this density related to the Radon-Nikodym derivative? Is it with respect to the Borel measure? Going back to the main question, can a PDF be expressed as an expectation? – W. Zhu Aug 23 '23 at 03:18
  • @W.Zhu I don't think that I understand your comment: you consider $X$ discrete rv and $x\mapsto \delta_X(x)$. Again, this is not a function, but rather a (random) distribution or a (random) measure [yes, that can be hard to wrap your head around]! In particular, it cannot be a Bernoulli rv, because Bernoulli rv are real-valued. – Mushu Nrek Aug 23 '23 at 08:25
  • @MushuNrek The Dirac delta is not a function, but the Kronecker delta is. The Kronecker delta at $X$ maps $x$ to $1$ if $X=x$ and $0$ otherwise, so its range is ${0,1}$. – W. Zhu Aug 24 '23 at 02:57
  • Agreed, but it is still not a Bernoulli rv, no? Indeed, it is, as a rv, function-valued. – Mushu Nrek Aug 24 '23 at 11:42
  • @MushuNrek We should say that they are point-wise expectations, e.g. the PMF of $X$ at any fixed $x\in\mathbf R$ is equal to the expectation of the indicator of $X=x$ (or equivalently the Kronecker delta of $x$ at $X$). – W. Zhu Aug 30 '23 at 06:35
  • @W.Zhu the probability mass functions has strictly no meaning when considering continuous random variables, because $\mathbb P(X = x) = 0$ for all $x$ in that case... – Mushu Nrek Aug 30 '23 at 15:17
  • @MushuNrek We were talking about the comment I made above. The statement about PMFs was made for discrete random variables, and the statement about CDFs was made for all random variables. – W. Zhu Aug 31 '23 at 14:31
  • @W.Zhu OK I see. If all your questions are answered now, consider to close the question. Otherwise, pls state precisely what you are unsure about. – Mushu Nrek Sep 01 '23 at 15:28
  • @MushuNrek Thank you for responding to my question, but as you see, the question has received too few responses, so I would rather leave it open. – W. Zhu Sep 05 '23 at 05:35
  • Related question: https://math.stackexchange.com/questions/54197/can-a-dirac-delta-function-be-a-probability-density-function-of-a-random-variabl – W. Zhu Sep 10 '23 at 09:39

0 Answers0