2

Is there a name for the class of functions whose derivative is expressed only in terms of the function value? One example is the exponential, another example is

\begin{align} s_1(t) = \frac{1}{1 + e^{-t}} \end{align}

with derivative

\begin{align} s'_1(t) = s_1(t)[1-s_1(t)]. \end{align}

Clarification

My question is related to writing about Neural Networks (NN). In neural networks you calculate the derivative of the output relative to the input by means of an algorithm called backpropagation, or backprop (which is really nothing but the chain rule expressed in a computationally-efficient manner).

An important computational advantage while doing backprop is to store the function value when propagating forward, and using that function value to compute the derivative when propagating backward. This is only possible if the derivative only depends on the function value (and not on, say, the variable value).

For example. Suppose that you have a working vector w:

# w is currently storing the value of t, the independent variable
w = [1, 2, 3]

in the first step you calculate the function value (you won't need the value of the independent variable $t$ anymore, so you overwrite the contents in memory)

# w is currently storing the value of s(t) = 1 / [1 + exp(-t)]
w = [0.7310585786300049, 0.8807970779778823, 0.9525741268224334]

in the next step you calculate the derivative value (you won't need the value of $s(t)$ anymore, so you overwrite the contents in memory)

# w is currently storing the value of s'(t) = s(t)[1-s(t)]
w = [0.19661193324148185, 0.10499358540350662, 0.045176659730912]

Notice that if $s'$ had a dependency on the value of $t$ (as opposed to only the value of $s(t)$) I would not be able to reuse the memory in w.

The specific paragraph I'm trying to improve reads as follows:

Pick an activation function whose derivative depends only on the function value, and not on the value of the independent variable. Such activation functions enable reusing the working memory during the backprop stage.

And I'd like it to know if this could be expressed more precisely: pick a ??? activation function.

Escualo
  • 163
  • They're solutions to autonomous differential equations, and if the graph of the function is a single piece you can characterize them as the (weakly) monotonic differentiable functions. I'm not aware of a better name. – epimorphic Feb 09 '17 at 18:46
  • @epimorphic My problem is that the differential equation is autonomous, bu what do you call the function itself? I'm trying to express the following idea: "a function whose derivative does not depend on the independent variable is called: ???". Or, put it another way: "Every ??? function can be written as an autonomous differential equation". – Escualo Feb 09 '17 at 18:53
  • I'm aware of that; my suggestions were "solutions to autonomous differential equations" and its variations, or in the certain aforementioned case, "(weakly) monotonic differentiable functions". If those don't do, you could always just define your own name in your work. I like the sound of "autonomously differentiable" (do note that it returns zero results on Google). – epimorphic Feb 09 '17 at 19:12
  • I like that name! It does seem to express the intent. If you add it as an answer I'll accept it. Thanks! – Escualo Feb 09 '17 at 19:13

2 Answers2

2

I think the answer to the literal question you ask is "no". I don't know of any name for that kind of function.

But behind that question, I think you are asking about describing a function as the solution to a (first order) differential equation, like $$ y' = Cy $$ for the exponential function or $$ y' = Cy(1-y) $$ for the logistic curve.

Solutions to first order equations often involve the exponential function.

Ethan Bolker
  • 95,224
  • 7
  • 108
  • 199
  • Thank you. My question is more related to being able to chain calculations and reuse storage (from a computer perspective). Probably the edit I added to the question can clarify? I regret that my question is very unclear - apologies. – Escualo Feb 09 '17 at 18:33
  • No need to apologize. Questions are often clarified here. @epimorphic 's comment is the best you'll get for a name for these functions. The wikipedia page he links to may help with numerical methods. – Ethan Bolker Feb 09 '17 at 18:53
1

I think what you are looking for is the term "ordinary differential equations". These are equations that are defined with functions of a single variable as well as the derivatives of those functions. Wolfram MathWorld has a nice overview: http://mathworld.wolfram.com/OrdinaryDifferentialEquation.html

Addison
  • 19