Constrained optimization in random variables

Question

I was handed this problem:

Let $X_1,\ldots, X_n$ be a sequence on $n$ independent random variables, with $\mathbb{E}\left[X_i\right]=\mu_i$ and $\mathbb{V}\left[X_i\right]=\sigma_i^2$. Find the constants $a_i$ so that $\mathbb{V}\left[\sum_{i=1}^2 a_i X_i\right]$ is minimized given that $\mathbb{E}\left[\sum_{i=1}^2 a_i X_i\right]=\mu$.

and I came up with the following solution:

It's true that $$\mathbb{E}\left[\sum_{i=1}^{n}a_{i}X_{i}\right]=\sum_{i=1}^{n}a_{i}\mathbb{E}\left[X_{i}\right]=\sum_{i=1}^{n}a_{i}\mu_{i}=\mu$$ and \begin{align} \mathbb{V}\left[\sum_{i=1}^{n}a_{i}X_{i}\right]&=\mathbb{E}\left[\left(\left(\sum_{i=1}^{n}a_{i}X_{i}\right)-\mathbb{E}\left[\sum_{i=1}^{n}a_{i}X_{i}\right]\right)^{2}\right]=\mathbb{E}\left[\left(\sum_{i=1}^{n}a_{i}X_{i}-\sum_{i=1}^{n}a_{i}\mu_{i}\right)^{2}\right]\\ &=\mathbb{E}\left[\left(\sum_{i=1}^{n}a_{i}\left(X_{i}-\mu_{i}\right)\right)^{2}\right]=\mathbb{E}\left[\left(\sum_{i=1}^{n}a_{i}\left(X_{i}-\mu_{i}\right)\right)\left(\sum_{i=1}^{n}a_{i}\left(X_{i}-\mu_{i}\right)\right)\right]\\ &=\mathbb{E}\left[\sum_{i=1}^{n}\sum_{j=1}^{n}a_{i}a_{j}\left(X_{i}-\mu_{i}\right)\left(X_{j}-\mu_{j}\right)\right]=\sum_{i=1}^{n}\sum_{j=1}^{n}a_{i}a_{j}\mathbb{E}\left[\left(X_{i}-\mu_{i}\right)\left(X_{j}-\mu_{j}\right)\right]\\ &=\sum_{i=1}^{n}\sum_{j=1}^{n}a_{i}a_{j}Cov\left(X_{i},X_{j}\right). \end{align} Since the random variables are independent, for $i\neq j\: : \: Cov\left(X_i,X_j\right)=0$ . So using Kronecker's delta we get $$\sum_{i=1}^{n}\sum_{j=1}^{n}a_{i}a_{j}Cov\left(X_{i},X_{j}\right)=\sum_{i=1}^{n}\sum_{j=1}^{n}a_{i}a_{j}\delta_{i,j}\mathbb{V}\left[X_{i}\right]=\sum_{i=1}^{n}a_{i}^{2}\sigma_{i}^{2}.$$ So we need to find constants $a_{i}$ so that the following minizes $\sum_{i=1}^{n}a_{i}^{2}\sigma_{i}^{2}$ given that $\sum_{i=1}^{n}a_{i}\mu_{i}=\mu$. We define the function $L:\mathbb{R}^{n}\to\mathbb{R}$ as $$L\left(a_{1},a_{2},\ldots,a_{n}\right)=\left(\sum_{i=1}^{n}a_{i}^{2}\sigma_{i}^{2}\right)+\lambda\left(\left(\sum_{i=1}^{n}a_{i}\mu_{i}\right)-\mu\right).$$ The critical points of $L$ are given by $$\frac{\partial L}{\partial a_{i}}=2\sigma_{i}^{2}a_{i}+\lambda\mu_{i}=0\Rightarrow a_{i}=-\frac{\lambda\mu_{i}}{2\sigma_{i}^{2}},\quad i=1,\ldots,n.$$ Replacing the $a_{i}$ we just found in the constraint we get $$\sum_{j=1}^{n}-\frac{\lambda\mu_{j}}{2\sigma_{j}^{2}}\mu_{j}=\mu\Rightarrow-\frac{\lambda}{2}\sum_{j=1}^{n}\frac{\mu_{j}^{2}}{\sigma_{j}^{2}}=\mu\Rightarrow\lambda=-\frac{2\mu}{\sum_{j=1}^{n}\frac{\mu_{j}^{2}}{\sigma_{j}^{2}}}.$$ We conclude the the critical point of $L$ are $$a_{i}=-\left(-\frac{2\mu}{\sum_{j=1}^{n}\frac{\mu_{j}^{2}}{\sigma_{j}^{2}}}\right)\frac{\mu_{i}}{2\sigma_{i}^{2}}=\mu\frac{\mu_{i}}{\sigma_{i}^{2}}\frac{1}{\sum_{j=1}^{n}\frac{\mu_{j}^{2}}{\sigma_{j}^{2}}},\quad i=1,\ldots,n.$$

But now I can't prove that that is a minimum. I thought that I could use the Cauchy - Schwarz inequality, but I'm kinda lost.

PS: Sorry for any and all grammatical errors I'm translating Greek.

Edit. Fix the mistakes pointed out by StubbornAtom.

Answer 1

This is a slight generalization of this question.

By Cauchy-Schwarz,

$$\sum_{i=1}^n (a_i\sigma_i)^2\sum_{i=1}^n \left(\frac{\mu_i}{\sigma_i}\right)^2\ge \left(\sum_{i=1}^n a_i\mu_i\right)^2$$

Or, $$\sum_{i=1}^n a_i^2\sigma_i^2\ge \mu^2\left(\sum\limits_{i=1}^n \frac{\mu_i^2}{\sigma_i^2}\right)^{-1}$$

Equality holds precisely when $a_i\sigma_i$ is proportional to $\frac{\mu_i}{\sigma_i}$ for every $i$, i.e. when $a_i=\frac{c\mu_i}{\sigma_i^2}$ for some non-zero constant $c$. From the constraint $\sum\limits_{i=1}^n a_i\mu_i=\mu$, we have $$c=\mu\left(\sum\limits_{i=1}^n \frac{\mu_i^2}{\sigma_i^2}\right)^{-1}$$

So the optimal solution is $$\hat a_i=\mu\left(\sum\limits_{i=1}^n \frac{\mu_i^2}{\sigma_i^2}\right)^{-1}\frac{\mu_i}{\sigma_i^2}\quad,\,i=1,\ldots,n$$

Answer 2

Another possible ad simpler solution is:

We observe that the second partial derivatives of $L$ are $$\frac{\partial^{2}L}{\partial a_{i}\partial a_{j}}=\begin{cases} 2\sigma_{i}^{2} & i=j\\ 0 & i\neq j \end{cases}.$$ So the Hessian matrix of $L$ will be $$D^{2}L=\begin{bmatrix}\frac{\partial^{2}L}{\partial a_{1}^{2}} & \frac{\partial^{2}L}{\partial a_{1}\partial a_{2}} & \cdots & \frac{\partial^{2}L}{\partial a_{1}\partial a_{n}}\\ \frac{\partial^{2}L}{\partial a_{2}\partial a_{1}} & \frac{\partial^{2}L}{\partial a_{2}^{2}} & \cdots & \frac{\partial^{2}L}{\partial a_{2}\partial a_{n}}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial^{2}L}{\partial a_{n}\partial a_{1}} & \frac{\partial^{2}L}{\partial a_{n}\partial a_{2}} & \cdots & \frac{\partial^{2}L}{\partial a_{n}^{2}} \end{bmatrix}=\begin{bmatrix}2\sigma_{1}^{2} & 0 & \cdots & 0\\ 0 & 2\sigma_{2}^{2} & \cdots & 0\\ \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & \cdots & 2\sigma_{n}^{2} \end{bmatrix}$$ We see that the eigenvalues of $D^2L$ are $\lambda_{i}=2\sigma_{i}^{2}\geq0,\,i=1,\ldots,n$. That means that $D^2L$ is semi-positive defined matrix, and as a result for $a_{i}=\mu\frac{\mu_{i}}{\sigma_{i}^{2}}\frac{1}{\sum_{j=1}^{n}\frac{\mu_{j}^{2}}{\sigma_{j}^{2}}},\,i=1,\ldots,n$ the value of $\mathbb{V}\left[\sum_{i=1}^{n}a_{i}X_{i}\right]$ is minimized.