Approximate $|x|$ with a smooth function

Question

I am trying to get the derivative of $|x|$, and I want that derivative function, say $g(x)$, to be a function of x.

So it really needs the |x| to be smooth (ex. $x^2$); I am wondering what is the best way to approximate |x| with something smooth?

I propose $\sqrt{x^2 + \epsilon}$, where $\epsilon = 10^{-10}$, but there should be something better? Perhaps Taylor expansion?

Sorry for any confusion. I should add some additional information here:

I want to use $|x|$ as part of an object function $J(x)$ which I want to minimize. So it would be nice to approximate $|x|$ with some smooth function so that I can get the analytic form of the first-order derivative of $J(x)$.

Thanks a lot.

Answer 1

A bit late to the party, but you can smoothly approximate $f(x) = |x|$ by observing $\partial f/\partial x = \mbox{sgn}(x)$ for $x \neq 0$. Therefore approximating the $\mbox{sgn}(x)$ function by

$$ f(x) = 2\mbox{ sigmoid}(kx)-1$$

(the $k$ being a parameter that allows you to control the smoothness), we get

$$ \partial f/\partial x = 2\left (\frac{e^{kx}}{1+e^{kx}} \right ) - 1 $$ $$ \Rightarrow f(x) = \frac{2}{k}\log(1+e^{kx})-x-\frac{2}{k}\log(2)$$

where the constant term was chosen to ensure $f(0) = 0 $.

Included are plots of the function for $x \in [-5,5]$, where $k=1$ is red, $k=10$ is blue and $k=100$ is black.

Note that if you wanted to use the more smooth $k=1$ in the interval $[-5,5]$ it may be worth applying a further linear transformation i.e. ~$5f(x)/3.6$ to ensure the values at the edge of the interval are correct.

Answer 2

I have used your $\sqrt{x^2+\epsilon}$ function once before, when an application I was working on called for a curve with a tight radius of curvature near $x=0$. Whether it is best for your purpose might depend on your purpose.

(Side note: The derivative of $|x|$ does not exist at $x=0$. In physics, we sometimes cheat and write ${d\over dx}|x| = \rm{sgn}(x)$, the "sign" function that gives $-1$ when $x<0$, $0$ when $x=0$, and $+1$ when $x > 0$. But only when we know that the value at $x=0$ will not get us into trouble!)

Answer 3

Another solution would be the function $x \rightarrow x \frac{e^{kx}}{e^{kx} + e^{-kx}} -x\frac{e^{-kx}}{e^{kx} + e^{-kx}} $ with higher the $k$, closer you will be to the absolute value function. Furthermore it is equal to 0 in 0 wich is sometimes a desirable feature (compared to the solution with the root square stated above). However you lose the convexity.

Answer 4

As you and others have mentioned, functions of the form $\sqrt{x^2 + 4\mu^2}$ can be a good approximation to $\left|x\right|$ and are standard in many optimization applications where the smoothing function meets certain bounding requirements. See, for example, the paper Optimality Conditions and a Smoothing Trust Region Newton Method for Non-Lipschitz Optimization where that function is used to develop a smoothing optimization algorithm.

Another smooth approximation that has been proposed in Fast Optimization Methods for L1 Regularization: A Comparative Study and Two New Approaches is

$$ \begin{equation} \mid x\mid \ \approx \frac{1}{\alpha} \left[ \log\left(1 + \exp(-\alpha x)\right) + \log\left(1 + \exp(\alpha x) \right)\right] \end{equation} $$ where $\alpha$ is taken to be a large positive number.

Answer 5

$$\forall x\neq0,\dfrac{d|x|}{dx} = \frac{x}{|x|} = \begin{cases} -1 & x<0 \\ 1 & x>0. \end{cases}$$