Lies, Damn Lies and,... Gradients?

Question

Help! I think I'm stuck in a local minimum and I can't get out! Ok that's not news, many people all over the world are stuck in local minima everyday. What is news is that in my case I know there is a path out, and I'm wondering if a gradient calculation will help me find it. Here is my situation,... Imagine a bowl-shaped surface centered at x = 0, y = 0. Now imagine that there is a path out of this bowl but that it lies perfectly on the diagonal (so where x = y). In this case I think the partial derivative in the x-direction will tell me that I'm at the bottom of a 'U' and that there is no where to go, whilst the partial derivative in the y-direction will also tell me the same. So in the end the gradient should also tell me that I'm stuck at the bottom of a bowl. Is this correct? In other words,... do gradients have blind spots? Is it true that the 'gradient' calculation is in essence just an approximation based on 'sample' partial derivatives taken in the respective directions of the coordinate system? So if I change the coordinate system I might very well get a different value for the gradient even for the same point on the surface? Or is there another type of 'gradient' that I can calculate which will tell me how to get out of the bowl regardless of coordinate system?

Thanks for your patience,...

Terry

Answer 1

The function $$ f(x, y) = [(x + y)(x - y)]^2 = (x^2 - y^2)^2 $$ has the property you've described: it's $U$-shaped when viewed in the $y = 0$ or $x = 0$ plane, but along the line $y = x$, it's horizontal, so there's a way to escape. It's also differentiable.

The real point is that "local min" implies "gradient is zero", but the other direction is not true at all (as $f(x, y) = x^2 - y^2 $ shows).

On the other hand, at places where the gradient is nonzero, it does tell you the direction of steepest descent...but only locally, and only to first order. If you stand on the slope of Bunker Hill in Boston, you can hardly expect the upward slope direction to point towards Mt. Everest, or even Mt. Washington.

Answer 2

The gradient of a scalar function at a point $(x_0,y_0)$ describes the change in the value of a function as you step away from the point in the maximal direction. Let's see how.

Say you have a scalar function describing a surface z = T(x,y). Then the gradient is given by

$$\nabla T(x,y) = \left(\frac{d}{dx}\hat{x} + \frac{d}{dy}\hat{y}\right)T(x,y) = \frac{dT(x,y)}{dx}\hat{x} + \frac{dT(x,y)}{dy}\hat{y}$$

At a specified point $(x_0,y_0)$, we have

$$\nabla T(x_0,y_0) = \frac{dT(x_0,y_0)}{dx}\hat{x} + \frac{dT(x_0,y_0)}{dy}\hat{y}$$

How can we intuitively interpret this result? Well, it's a vector. It has two components, one giving a magnitude in the $x$ direction and another pointing in the $y$ direction. As a hiker using $(x,y)$ coordinates in the plane, you already like this. Could the gradient guide us out of the valley? Well, we need to know more. The thing is, we need to understand what the direction of gradient means.

Consider an arbitrary unit vector $\hat{v}$ and summon the definition of the dot product:

$$\hat{v}\cdot\nabla T = |\hat{v}||\nabla T|cos\theta = |\nabla T|cos\theta$$

If our arbitrary unit vector is parallel to the gradient, then $\theta = 0$ and the gradient is maximized.

$$\hat{v}\cdot\nabla T = |\nabla T|$$

So if we follow the gradient, we find the steepest incline or cough grad(i)e(nt). It tells us how quickly the walls of the metaphorical valley rise up in these two orthogonal directions, as we march in the maximal direction, and it does so by considering an infinitesimal change in either direction... the slightest nudge.

Now, if I elect to travel in a different direction , then I'm taking a directional derivative. Dotting this new direction to the gradient will tell me what sort of path I may encounter.

Answer 3

The gradient tells you the path of steepest ascent/descent at the exact point where you are. The reason we are concerned about local minima vs. global minima is precisely that when you have a small "bump" or "dent" in a continuous multidimensional function plot, the gradient in the immediate neighborhood of the bump or dent will point toward the bump or dent and not toward a high mountain or deep valley that might exist elsewhere.

Imagine a round stone sitting high up on a hillside. The stone stays where it is because it is in a very shallow hole whose sides slope inward, so if you moved the stone a tiny bit in any direction it would just fall back to the bottom of the hole. But if you push the rock far enough in the right direction, you will push it out of the hole and it may roll all the way to the bottom of the hill.

At the very bottom (or top) of a local minimum (or maximum), the gradient is zero and does not tell you to go in any direction. You can also have a zero gradient that is neither a minimum nor a maximum: an inflection point or a saddle point.

In another answer, you have a function $f(x,y)=(x^2-y^2)^2$ which has actually a global minimum along the lines $y=x$ and $y=-x$. (The minimum is not unique, but it is global because there is no lower point anywhere on the plot of that function.) If you happen to be sitting at $(0,0)$, the gradient will be zero, giving you no clue which way to go. So try movements in "all" directions--or since you can't try all the infinitely many possible angles, some reasonable set of angles, but more than just the four directions given by "increase/decrease $x$" and "increase/decrease $y$".

If you don't always go in only the positive or negative $x$ or $y$ directions, and you happen to be a saddle point instead of actually at either a global or local minimum, for example, if your function were actually $g(x,y) = (x^2 - y^2)^2 - \frac12(x^4 + y^4)$ (see this function's three-dimensional plot) you would have a better chance of finding your way toward a minimum.

Note that for the function $g(x,y)$, the gradient at $(0,0)$ is zero even though the graph of the function drops off in four directions. (In four other directions it rises.) That's an example of a saddle point (though the usual examples drop off in just two directions).

Recall that even in two dimensions you can have a zero gradient (which in two dimensions is just a derivative) at a point that is not a minimum. One way of course is to be at a maximum, but you can also be at inflection point such as the point at $x=0$ in the graph of $y=x^3$. In three dimensions, the graph of $h(x,y) = -(x+y)^3$ has an "inflection" along the line $y=-x$ where the gradient is zero, but there clearly is neither a maximum nor minimum there.

The function $k(x,y) = (x-y)^2 - (x+y)^3$ (see its graph here) has a zero gradient at $(0,0)$, but slopes down everywhere else on the line $y=x$ as $x$ and $y$ increase.

It's even possible to have a "ledge" in a function where the gradient is zero over some region, but the function increases as you leave that region in some directions but decreases in others. There is a two-dimensional version of this, too.

So, just as in two dimensions, merely examining the gradient at one point does not tell you whether you're at a local minimum, nor does it always tell you which direction to go if you want to decrease the function. But you can still get information by looking at the gradients at multiple points. If you are at a local minimum, the gradients at all nearby points will point away from the bottom of the bowl (meaning you would go toward the bottom of the bowl to decrease the function from those points). If you are at a saddle point or some kind of inflection point, however, you will find some nearby gradients point away from that point and some point toward it. That's a clue that you are not at a minimum, and also a clue about which way to go in order to decrease the function value.