2

Suppose that we have the following (unconstrained) optimization problem:

$$ \min_{\mathbf{w},b} \frac{1}{2}\Arrowvert\mathbf{w}\Arrowvert^2 + C\sum_{i=1}^{l}g_i(\mathbf{w},b), $$

where $\mathbf{w}\in\mathbb{R}^n$, $b\in\mathbb{R}$. Moreover, for $i=1,\dots,l$, the functions $g_i: \mathbb{R}^n\times \mathbb{R} \to \mathbb{R}$ are differentiable and their partial derivatives with respect to $\mathbf{w}$, $b$ are known.

Which optimization methods should I use for that end? Is there a specially appropriate technique for the specific objective function?

Any useful comment/advice will be fully appreciated! Thanks a lot!

nullgeppetto
  • 3,006

1 Answers1

1

If also the second derivatives of the $g_i$ are known, you may want to use a (globalized) Newton method. Otherwise, a quasi-Newton method is a good choice.

gerw
  • 31,359
  • thanks @gerw! But what I have found so far concerns the case of optimizing with respect to one variable, say $\mathbf{x}$ (in our case $\mathbf{w}$). Then, what happens when I want to have a joint optimization; with respect both to $\mathbf{w}$ and $b$? – nullgeppetto Dec 11 '13 at 11:24
  • Most literature on optimization deals with the minimization w.r.t. vectors. And $(w,b) \in \mathbb{R}^{n+1}$ is indeed a vector. – gerw Dec 11 '13 at 12:20
  • thanks. Yeah, that's makes sense... I was looking for something more "explaining", which might include some equations, but ok, I'll try to find out what's going on! Thanks again! If you want to contribute with some eqs, it'll be welcome too! – nullgeppetto Dec 11 '13 at 12:27