The context of my question is the kernel ridge regression (in the field of Statistics). The question is to find a closed formula for $\lambda^{*}$ \begin{align*} \lambda^{*}=\underset{\lambda >O }{\text{argmin}}~\hat{R_{p}}(\lambda) \end{align*} where $\hat{R_{p}}(\lambda)$ is defined by, $\forall \lambda >0$, \begin{align} \hat{R_{p}}(\lambda)=\frac{1}{p{n \choose p}}\sum_{e}\sum_{i\in \overline{e}}(Y_{i}-\hat{f_{\lambda}^{e}}(x_{i}))^{2}. \end{align} where $n\in\mathbb{N}^{*},1\leq p \leq n$ ; $(x_{i},Y_{i})_{i\in\{ 1, \ldots, n\}}$ is the sample ; e$\subset\{ 1, \ldots, n\}$ with cardinality $n-p$ ; e and $\overline{e}$ are respectively the training sample and the test sample ($\overline{e}$ is the complementary of $e$); $K^{i,e}$ is a $1\times\text{card(e)}$ vector $(k(x_{i},x_{j}))_{j\in e}$ for $i\in\{ 1, \ldots, n\}$ fixed ; $K^{e,e}$ is a $card(e)\times card(e)$ matrix $(k(x_{i},x_{j}))_{(i,j)\in e\times e}$ ; $\hat{f_{\lambda}^{e}}(x_{i})=K^{i,e}(K^{e,e}+\lambda I_{n-p})^{-1}Y_{e}$.
I should precise that $K^{e,e}$ is positive semi-definite symmetric matrix because the Gram matrix $K=(k(x_{i},x_{j}))_{(i,j)\in\{ 1, \ldots, n\}^{2}}$ is a positive semi-definite symmetric matrix. My Supervisor give me the idea to use : \begin{align*} (I-M)^{-1}=\sum_{k=0}^{+\infty}M^{k} \quad \text{ valid if }\quad||M||_{2}<1. \end{align*} Thus, we use the approximation valid if $\lambda>||K||_{2}$ \begin{align*} (K^{e,e}+\lambda I)^{-1}&=\lambda^{-1}(I+\lambda^{-1}K^{e,e})^{-1}\\ &\approx\sum_{k=0}^{N}(-\lambda^{-1}K^{e,e})^{k}. \end{align*} Replacing $(K^{e,e}+\lambda I)^{-1}$ by $\displaystyle \sum_{k=0}^{N} (-\lambda^{-1}K^{e,e})^{k}$ leads to an approximation of $\hat{R_{p}}(\lambda)$ denoted by $\hat{R_{p}^{N}}(\lambda)$. Then, instead of finding out a closed formula for $\lambda^{*}$, we hope to find a solution to for a fixed $N\in\mathbb{N}^{*}$ \begin{align*} \lambda_{N}^{*}=\underset{\lambda >O }{\text{argmin}}~\hat{R_{p}^{N}}(\lambda) \end{align*} I manage to compute a closed formula for $\hat{R_{p}^{0}}$ and $\hat{R_{p}^{1}}$ but the problem that I meet is that the approximation is valid for $\lambda > ||K||_{2}$ and not for $\lambda\in\mathbb{R}_{+}^{*}$. My question is to find out a closed formula for $\lambda^{*}$ or at least $\hat{R_{p}}$.
