1

Here is a sum of absolute values

$$V(x)=\sum_{i=1}^n|x_i - x_{i-1}|$$

and the sum of square errors is

$$U(x)=\sum_{i=1}^n(x_i - a_i)^2$$

My problem is to minimize $V(x)$ while keeping $U(x)$ as small as possible, so I might formulate a new objective function, e.g.,

$$L(x) = V(x) + \lambda U(x)$$

and try to minimize $L(x)$, right? If so, how can I solve it?

avocado
  • 1,209

1 Answers1

4

This is a one dimensional TV denoising problem. In the 2D case you are denoising an image. This problem has received much attention and there are many good approaches. A simple approach that is at least close to state of the art is to use the Pock-Chambolle algorithm, described in the paper "A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging" by Chambolle and Pock. Another approach is to formulate the dual problem and solve it using FISTA. You could also reformulate the problem as a quadratic program and solve it using CVX or with some other implementation of an interior point method.

By the way, in your definition of $V(x)$, shouldn't $i$ start at $2$?

Edit: adding more details.

Perhaps the easiest approach to understand is to solve the dual problem using projected gradient ascent. This is a nice exercise in deriving dual problems.

Let's first rewrite our problem using more concise notation:

\begin{equation*} \text{minimize}_{x \in \mathbb R^n} \quad \| Dx \|_1 + \frac{\rho}{2} \| x - a \|_2^2 \end{equation*} where $D$ is an $n \times n$ forward difference matrix. (Note that the last row of $D$ is all zeros.)

This problem is equivalent to \begin{align*} \text{minimize}_{x,y} &\quad \| y \|_1 + \frac{\rho}{2} \|x - a \|_2^2 \\ \text{subject to} & \quad y = Dx. \end{align*}

The Lagrangian is \begin{align*} L(x,y,z) &= \| y \|_1 + \frac{\rho}{2} \| x - a \|^2 + \langle z, y - Dx \rangle \\ &= \| y \|_1 + \langle z,y \rangle + \frac{\rho}{2} \| x - a \|^2 - \langle D^Tz, x \rangle . \end{align*}

The dual function $g(z)$ is obtained by minimizing $L(x,y,z)$ with respect to $x$ and $y$. Minimizing with respect to $y$, we find that $g(z) = -\infty$ unless $\|z\|_{\infty} \leq 1$. By completing the square, we could work out explicitly what we get when we minimize with respect to $x$. We would find that the dual problem requires that we maximize a quadratic function of $z$, subject to the constraint that $\| z \|_{\infty} \leq 1$. This can be done using (projected) gradient ascent. (Or we could use FISTA.)

littleO
  • 51,938
  • Yes, you're right. I kinda have hard time understanding Chambolle's approach, so I post it here hoping someone can give me a detailed answer, :-) – avocado Jul 17 '13 at 06:04
  • Out of curiosity, how did you get interested in this problem? – littleO Jul 17 '13 at 06:10
  • I'm learning Computer Vision stuff, and the CV book talks about TV denoising technique without lecturing much detail about it, so googled about it and found the Chambolle's approach, that's it. Thank you so much, from your profile, you're totally an export of math. – avocado Jul 17 '13 at 07:01
  • @littleO, Great answer. Could you elaborate a little more on how to work on the Dual Problem and derive the form to utilize FISTA? Thank You. – Royi May 31 '16 at 17:49