8

I'm kind of rusty in calculus.

Why is the ReLU function not differentiable at $f(0)$?

$$ f(x) = \begin{cases} 0 & \text{if $x \leq 0$} \\ x & \text{if $x > 0$}. \end{cases} $$

wchargin
  • 1,802
rvimieiro
  • 185
  • 2
    You should say “differentiable at $0$”, not “differentiable at $f(0)$”. (Not that it actually matters in this case, since $f(0)$ happens to be $0$, but anyway...) – Hans Lundmark Aug 08 '19 at 07:29

2 Answers2

14

Because it has a sharp corner at $0$, so it doesn't have a well defined tangent line; think about it, you can imagine many lines going through $(0, 0)$ that are tangent to the graph, so there are many possible tangent lines.

More formally, we have to investigate the limit

$$\lim_{h \to 0} \dfrac{f(0+h) - f(0)}{h}$$

This limit does not exist for the function, because if you let $h$ approach $0$ from the right, you get

$$\lim_{h \to 0^+} \dfrac {h-0}{h}=1.$$

While if you let $h$ approach $0$ from the left, that limit

$$\lim_{h \to 0^-} \dfrac {0-0}{h}=0.$$

Therefore the limit does not exist, so the function is not differentiable at $0$.

Ovi
  • 23,737
  • "a tangent line is one which touches the graph only at one point" isn't entirely true. Consider the tangent of $x^2(x-1)$ at $x = 0$. It intersects the graph at $x = 1$. – Arthur Aug 08 '19 at 06:55
  • I would not call that sharp corner or bend at $0$ a "cusp". A cusp is when the curve kind of turns 180 degrees so that the incoming tangent vector has the opposite direction of the outgoing tangent vector. The function $x\mapsto\sqrt{|x|}$ has a cusp. – Jeppe Stig Nielsen Aug 08 '19 at 07:27
  • @Arthur Yeah I'm aware, but I think for the purposes of the OP it's fine. But I'm not actually sure of the definition of tangent line myself; the best thing I could think of is: "L is tangent to $f$ at $x$ if there exists some neighborhood around $x$ in which $L$ is either completely below or completely above the graph of $f$." This definition seems satisfactory to me from a geometric point of view; but then we would have to give up the notion that "derivatives give tangent lines," because for example $x^2 \sin (\frac 1x).$ – Ovi Aug 08 '19 at 13:44
  • The tangent of $x^3$ at $x = 0$ then becomes problematic. I don't think you can get a proper definition of tangent without going into derivatives, or at least the same kinds of limits as the ones that define derivatives. – Arthur Aug 08 '19 at 13:47
  • @JeppeStigNielsen Oh thanks for the info, I wasn't aware; my highschool teacher told us all of these sharp corners are cusps. I have fixed the answer. – Ovi Aug 08 '19 at 13:47
  • @Arthur But if you define tangent line using derivative, then I think it becomes problematic with $x^2 \sin (\frac 1x)$ because (to me) that doesn't seem like it should be called a tangent line. I was giving priority to the geometric definition because the notion of tangent line is much older than the (analytic) idea of derivative. – Ovi Aug 08 '19 at 13:49
  • @Arthur Oh but my definition wouldn't work with vertical tangent lines lol; I guess I would have to add "or is completely to the left or completely to the right" too. – Ovi Aug 08 '19 at 13:51
  • For the case where the tangent "crosses" from lying on one side (say the lower side) to the other side of the curve, like $x\mapsto x^3$ mentioned above, see also inflection point (Wikipedia). – Jeppe Stig Nielsen Aug 08 '19 at 16:10
6

If you look at $x > 0$, or the righthand derivative, $$\frac{df}{dx} = \frac{d}{dx} x = 1$$ for all $x$.

If you look at $x \le 0$, or the lefthand derivative, $$\frac{df}{dx} = \frac{d}{dx} 0 = 0$$ for all $x$.

Since $x = 0$ is the "break" point, the lefthand and righthand derivatives are not the same, and thus, the derivative is not defined at $x = 0$.

Varun Vejalla
  • 8,873
  • 3
    The derivative can exist without being continuous. You have argued that $f$ can't be continuously differentiable, but how about differentiable? – Joonas Ilmavirta Aug 08 '19 at 07:21
  • I'm saying that the left and right hand derivatives are not the same at $x=0$. Therefore, the derivative does not exist. I'm not really saying anything about continuity. – Varun Vejalla Aug 08 '19 at 14:48
  • @JoonasIlmavirta Coming back to this answer after more than a year, could you expand on what you meant? I think I had misunderstood your comment before. – Varun Vejalla Aug 22 '20 at 20:56
  • What you computed is $\lim_{x\to0\pm}f'(x)$, whereas the one-sided derivatives at zero are $\lim_{h\to0\pm}h^{-1}[f(h)-f(0)]$. These are not the same thing! A function can be everywhere non-differentiable but it can still have one or both one-sided derivatives at zero. You observed that $f$ is differentiable whenever $x\neq0$ and that the limits at $x=0$ disagree. This only shows that if $f$ is differentiable everywhere, it cannot be continuous, unless you are implicitly appealing to a theorem that bridges the gap for you. – Joonas Ilmavirta Aug 23 '20 at 08:46
  • Ok, I think I understand now. I will update my answer accordingly. – Varun Vejalla Aug 23 '20 at 17:07
  • You can also find the computation of left and right derivatives at zero and the conclusion that the derivative cannot exist in Ovi's answer. – Joonas Ilmavirta Aug 23 '20 at 18:46