2

Consider the product $\left|w\sigma'(wa+b)\right|$, where $\sigma(x) = \frac{1}{1+e^{-x}}$. Suppose that $|w| \geq 4$, and $|w\sigma'(wa+b)|\geq 1$. Show that the set of $a$ satisfying that constraint can range over an interval no greater in width than

$$\frac{2}{|w|}\ln\Bigg( \frac{|w|(1+\sqrt{1−4/|w|}}{2}−1\Bigg).$$

Not quite sure where to start, I tried to work with the original equation:

$$e^{wa+b}/\left(e^{wa+b}+1\right)^2 \geq 1/|w| \implies wa+b - 2\ln\left(e^{wa+b}+1\right) \geq -\ln(w).$$

However, this doesn't seem to be going anywhere, and I'm just not sure how to go about finding the width of an $\textit{interval}$. Appreciate any suggestions/solutions, thanks!

Davide Giraudo
  • 172,925
b_pcakes
  • 1,501

1 Answers1

2

Starting with

$$\lvert w\sigma'(wa+b)\rvert \geq 1$$

Using the definition of the sigmoid and its derivative,

$$\lvert w\rvert \frac{e^{wa+b}}{(1+e^{wa+b})^2} \geq 1$$

Let $u \equiv e^{wa+b}$. Then,

$$\lvert w\rvert \frac{u}{(1+u)^2} \geq 1$$

Rearranging,

$$u^2+(2-\lvert w\rvert)u+1 \leq 0$$

Solving for $u$ gives

$$\frac{(\lvert w\rvert-2)-\sqrt{w^2-4\lvert w\rvert}}{2} \leq u \leq \frac{(\lvert w\rvert-2)+\sqrt{w^2-4\lvert w\rvert}}{2}$$

This is where the interval comes from. Before we get the width of that interval, we must finish solving for $a$. Note that we know both interval bounds are real-valued because $\lvert w \rvert \geq 4$. Plugging in for the definition of $u$ and doing some rearranging to better match the answer,

$$\frac{\lvert w\rvert\left(1-\sqrt{1-4/\lvert w\rvert}\right)}{2}-1 \leq e^{wa+b} \leq \frac{\lvert w\rvert\left(1+\sqrt{1-4/\lvert w\rvert}\right)}{2}-1$$

From here you can quickly solve for $a$:

$$\frac{1}{\lvert w\rvert}\ln\left(\frac{\lvert w\rvert\left(1-\sqrt{1-4/\lvert w\rvert}\right)}{2}-1\right) - \frac{b}{\lvert w\rvert}\leq a \leq \frac{1}{\lvert w\rvert}\ln\left(\frac{\lvert w\rvert\left(1+\sqrt{1-4/\lvert w\rvert}\right)}{2}-1\right)-\frac{b}{\lvert w\rvert}$$

So you can see $a$ lies on that interval. To get the width of the interval, simply subtract the lower bound from the upper bound:

$$\left[ \frac{1}{\lvert w\rvert}\ln\left(\frac{\lvert w\rvert\left(1+\sqrt{1-4/\lvert w\rvert}\right)}{2}-1\right)-\frac{b}{\lvert w\rvert}\right]-\left[\frac{1}{\lvert w\rvert}\ln\left(\frac{\lvert w\rvert\left(1-\sqrt{1-4/\lvert w\rvert}\right)}{2}-1\right) - \frac{b}{\lvert w\rvert}\right]$$

$$=\frac{1}{\lvert w\rvert}\ln{\left(\frac{\lvert w\rvert\left(1+\sqrt{1-4/\lvert w\rvert}\right)-2}{\lvert w\rvert\left(1-\sqrt{1-4/\lvert w\rvert}\right)-2}\right)}$$

Notice we lose the dependence on $b$. Multiplying the numerator and denominator inside the logarithm by $\lvert w\rvert\left(1+\sqrt{1-4/\lvert w\rvert}\right)-2$ and simplifying:

$$=\frac{1}{\lvert w\rvert}\ln{\left(\left[\frac{\lvert w\rvert\left(1+\sqrt{1-4/\lvert w\rvert}\right)}{2}-1\right]^2\right)}$$

$$=\frac{2}{\lvert w\rvert} \ln{\left(\frac{\lvert w\rvert(1+\sqrt{1-4/\lvert w\rvert})}{2}-1\right)}$$

This is the width of the interval. From our solution to $a$, you can see that the interval is centered about $-b/\lvert w\rvert$.

Brenden Petersen
  • 141