Since $f(x) = (x-b)^2 + (1 - b^2)$, the vertex will be at $(b, 1-b^2)$.
Also $f(1) = 2-2b$ and $f(0) = 1$.
When the vertex is at or to the left of $x=0$, then $f(x)$ is increasing on $[0, 1]$. So
$$ \max_{x \in [0,1]}f(x) - \min_{x \in [0,1]}f(x) = f(1) - f(0) = 1-2b$$
and $1-2b=4$ when $b = -\frac 32$
When the vertex is at or to the right of $x=1$, then $f(x)$ is decreasing on $[0, 1]$. So
$$ \max_{x \in [0,1]}f(x) - \min_{x \in [0,1]}f(x) = f(0) - f(1) = 2b-1$$
and $2b-1=4$ when $b = \frac 52$
When the x-coordinate of the vertex is in the interval $[0,\frac 12]$ then
$\begin{align}
\max_{x \in [0,1]}f(x) - \min_{x \in [0,1]}f(x)
&= \max\{f(0),f(1)\} - (1-b^2)\\
&= f(1) - (1-b^2)\\
&= (b-1)^2
\end{align}$
This does not equal $4$ when the x-coordinate of the vertex, b, is in the interval $[0,\frac 12]$
Finally, when the x-coordinate of the vertex is in the interval $(\frac 12, 1]$ then
$\begin{align}
\max_{x \in [0,1]}f(x) - \min_{x \in [0,1]}f(x)
&= \max\{f(0),f(1)\} - (1-b^2)\\
&= f(0) - (1-b^2)\\
&= b^2
\end{align}$
This does not equal $4$ when the x-coordinate of the vertex, b, is in the interval $(\frac 12, 0]$.
So the requested sum is -$\frac 32 + \frac 52 = 1$