Can anyone shed some light on the below statistical theory questions?

Question

Can anyone shed some light on the below:

1. Consider a set with $N$ distinct members, and a function $f$ defined on $\mathbb Q$ that takes the values $0$, $1$ such that $\frac1N\sum_{x\in\mathbb Q} f(x) = p$. For a subset $S$ of $\mathbb Q$ of size $n$, define the sample proportion $$p = p(S) = \frac1N\sum_{x\in S} f(x)$$ If each subset of size $n$ is chosen with equal probability, calculate the expectation and standard deviation of the random variable $p$.

2. 1. Let $X\sim \mathcal N(0, 1)$ be a normally distributed random variable with mean 0 and variance 1. Suppose that $x \in \mathbb R, x > 0$. Find upper and lower bounds for the conditional expectation $E(X \mid X >x)$
   2. Now suppose that $X$ has a power law distribution, $P(X >x) = ax^{-b}$, for $x>x_0>0$, and some $a> 0, b> 1$. Calculate the conditional expectation $E(X\mid X>x), x >x_0$

Many thanks in advance.

Answer 1

Regarding the upper limit of $E(x|x>X)$ when x is Normal distributed, a simple and more stringent value is

$E(x|x>X)\leq X+\sqrt{2/\pi}$.

Just looking at the Gaussian function one can infer that the difference

$E(x|x>X)-X$ decreases with $X$, so that it will have its maximum value ($\sqrt{2/\pi}$) when $X=0$. Thus for all $X\geq0$ one has $E(x|x>X)-X \leq E(x|x>0)$. It follows that

$E(x|x>X) \leq X + (E(x|x>0)) \rightarrow E(x|x>X) \leq X + \sqrt{2/\pi}$.

I just plotted the result and it works better than the one derived from the Abramowitz and Stegun limit, for all $X\geq 0$.

A trivial lower limit is of course $E(x|x>X)\geq X$

Answer 2

Taking on the second question.. $$ E(X|X>x) = \int_{x}^{\infty}tP(t,X>x)dt = \int_{x}^{\infty}t\frac{P(t)}{P(X>x)}dt\tag{1} $$ where $$ P(x) = \mathcal N(0, 1) = \frac{1}{\sqrt{2\pi}}\mathrm{e}^{-\frac{x^2}{2}} $$ In Eq.1 I use dummy variables to lessen the confusion. $$ E(X|X>x) = \frac{1}{\sqrt{2\pi}}\frac{1}{P(X>x)}\int_{x}^{\infty}t\mathrm{e}^{-\frac{t^2}{2}}dt $$ the integral is trivial results in $$ E(X|X>x) =\frac{1}{\sqrt{2\pi}}\frac{1}{P(X>x)}\mathrm{e}^{-x^2/2} $$ remembering that $$ P(X>x) = \frac{1}{\sqrt{2\pi}}\int_x^{\infty}\mathrm{e}^{-t^2/2}dt, $$ which is also equal to $\frac{1}{2}\mathrm{erfc}(x/\sqrt{2})$

From Abramowitz and Stegun you can obtain an upper and lower bound to the erfc function as $$ \sqrt{\frac{2}{\pi}}\frac{\mathrm{e}^{-x^2/2}}{x+\sqrt{x^2+4}}< \frac{1}{2}\mathrm{erfc}(x/\sqrt{2}) < \sqrt{\frac{2}{\pi}}\frac{\mathrm{e}^{-x^2/2}}{x+\sqrt{x^2+\frac{8}{\pi}}} $$ and further more $$ \sqrt{\frac{\pi}{2}}\frac{ x+\sqrt{x^2+\frac{8}{\pi}}}{\mathrm{e}^{-x^2/2}}< \frac{1}{\frac{1}{2}\mathrm{erfc}}< \sqrt{\frac{\pi}{2}}\frac{ x+\sqrt{x^2+4}}{\mathrm{e}^{-x^2/2}} $$ putting it all together we find $$ \frac{x+\sqrt{x^2+\frac{8}{\pi}}}{2} < E(X|X>x) < \frac{x+\sqrt{x^2+4}}{2} $$

$\textbf{Part 2}$ We have to determine the probability of $P(x)$ first to be able to compute the conditional expectation. $$ P(x) = \frac{\alpha-1}{x_0}\left(\frac{x}{x_0}\right)^{-\alpha} $$ here $x_0$ is the min value as outlined in the OP. $$ \int_x^{\infty}\frac{\alpha-1}{x_0}\left(\frac{t}{x_0}\right)^{-\alpha}dt =ax^{-b} $$ we find that $$ \alpha = b+1,\\ \left(\frac{1}{x_0}\right)^{1-\alpha} = a. $$ this yields that the original distribution should be $$ P(x) = abx^{-(b+1)}. $$

Then to compute $$ \begin{eqnarray} E(X|X>x) &=& \frac{1}{P(X>x)}\int_{x}^{\infty}t\left(abt^{-(b+1)}\right)dt \\ &=& \frac{ab}{ax^{-b}}\int_x^{\infty}t^{-b}dt \\ &=& \frac{b}{x^{-b}}\frac{x^{1-b}}{b-1}\\ &=& \frac{b}{b-1}x \end{eqnarray} $$

Answer 3

The way I see it, $f \left( x \right)$ follows a Bernoulli distribution with success probability $p$. Assuming this is correct, then the distribution of $n \hat{p}$ given $S$ would be a binomial random variable with success probability $p$ and $n$ trials. \begin{equation} \left. n \hat{p} \middle| S \right. \sim \mathcal{B} \left( p , n \right) \text{.} \end{equation} The conditional mean and variance of $n \hat{p}$ are \begin{align*} \mathrm{E} \left[ n \hat{p} \middle| S \right] = & n p \text{,} \\ \mathrm{Var} \left[ n \hat{p} \middle| S \right] = & n p \left( 1 - p \right) \text{,} \end{align*} and from the properties of the expectation it follows that \begin{align*} \mathrm{E} \left[ \hat{p} \middle| S \right] = & p \text{,} \\ \mathrm{Var} \left[ \hat{p} \middle| S \right] = & \frac{p \left( 1 - p \right)}{n} \text{.} \end{align*} Now it's time to marginalize out $S$, \begin{align*} \mathrm{E} \left[ \hat{p} \right] = & p \text{,} \\ \mathrm{Var} \left[ \hat{p} \right] = & p \left( 1 - p \right) \sum_{n = 1}^N \frac{P \left( n \right)}{n} \text{,} \end{align*} where $P \left( n \right)$ is $n$'s probability mass function. Since $S$ is chosen with equal probability, and the number of distinct $n$-element subsets in an $N$-element set such as $\Omega$ is \begin{equation*} \left( \begin{array}{c} n \\ N \end{array} \right) = \frac{N!}{n! \left( N - n \right)!} \text{,} \end{equation*} I'd be tempted to say that $P \left( n \right)$ is the reciprocal of this coefficient. But this doesn't lead anywhere.