1

I'm reading through http://ufldl.stanford.edu/tutorial/supervised/LinearRegression/

enter image description here

How can I satisfy myself that the highlighted statement is true?

Can anyone Point me towards an intuitive understanding or visualisation?

PS Links:
https://en.wikipedia.org/wiki/Gradient#Differential_or_.28exterior.29_derivative -- doesn't seem to help.
Gradient, tangents, planes, and steepest direction
https://en.wikipedia.org/wiki/Directional_derivative

Community
  • 1
P i
  • 2,136

2 Answers2

3

By the chain rule, the directional derivative of $f$ in the direction $\mathbf{v}$ is $\nabla f\cdot\mathbf{v}=\|\nabla f\|\|\mathbf{v}\|\cos\theta=\|\nabla f\|\cos\theta$ (because a direction vector is a unit vector), where $\theta$ is the angle between the gradient and the direction vector.

This is maximized precisely when $\mathbf{v}$ points in the direction of $\nabla f$ (so that $\cos\theta=1$), and the resulting maximum is

$$\nabla f\cdot\frac{\nabla f}{\|\nabla f\|}=\|\nabla f\|^2/\|\nabla f\|=\|\nabla f\|$$

symplectomorphic
  • 18,604
1

This is because the directional derivative $D_uJ(\theta)$ in the direction of the unit vector $u$ is equal to $$D_uJ(\theta)=\langle\nabla F(\theta),u\rangle,$$ and the inner product is maximal when the vectors are collinear, with a positive coefficient of proportionality, as is known from the equality case in Cauchy-Schwarz inequality.

Bernard
  • 175,478
  • My answer already says this. And collinearity is necessary but not sufficient for maximization (rather than for extremization). $\nabla F$ and $-\nabla F$ are collinear, but one gives the maximum and the other gives the minimum. – symplectomorphic Aug 18 '16 at 12:53
  • You posted when I was writing, and I didn't look if an answer had been posted meanwhile. For the other point, true, I forgot the condition on the collinearity coefficient. – Bernard Aug 18 '16 at 13:04