Suppose I have a function $f(x)$ for which I want to find minimums.
I understand that differentiation with respect to $x$ will give direction $+/-$ in $x$ axis to follow in order to minimize.
Choosing a fixed step is understandable, but why do we update by $x'=x - f'(x)\cdot\text{step}$?
$f'(x)$ is the slope and is relevant to $y$ axis.
A more understandable procedure to me is this update $x'=x -\text{sign}(f'(x))\cdot\text{step}$.
Thnaks.
joseph.
The algorithms of steepest descent rely on the observation that on a neighbourhood of a point x the function f decrease the most strongly in the oppositive direction of the gradient, $\nabla f(x)$.
The direction $-\nabla f(x)$ is a descent direction of f in x, as $f'(x).d=-||\nabla f(x)||^2<0$, so for a step $\alpha$ sufficiently small $$ f(x - \alpha \nabla f(x) < f(x) $$
Then, if you take a direction of the form $-sign(\nabla f(x))$ it won't be a descent direction anymore.
