You can add terms in the exponent to cancel higher derivatives at the origin. They need to be antisymmetric, and they shouldn't mess up the behaviour at infinity. Powers of $\sin x$ satisfy those requirements. To cancel the third derivative:
$$
\frac1{1+\exp\left(-x-\frac1{12}\sin^3x\right)}=\frac12+\frac x4+O\left(x^5\right)\;.
$$
To cancel the fifth derivative:
$$
\frac1{1+\exp\left(-x-\frac1{12}\sin^3x-\frac{13}{240}\sin^5x\right)}=\frac12+\frac x4+O\left(x^7\right)\;.
$$
And so on. Here's a plot of the second version.
This comes at a price – forcing the function into an "unnatural" linearity at the origin while keeping it analytic causes slight oscillations elsewhere. Of course, if you don't care about analyticity, you should just splice a linear function into a sigmoid.
P.S.: I was assuming that you want to keep the asymptotic behaviour roughly unchanged. If you don't care about that, you can do it more simply:
$$
\frac1{1+\exp\left(-x-\frac1{12}x^3\right)}=\frac12+\frac x4+O\left(x^5\right)\;.
$$
The plot looks a lot like what you wanted, but it converges to the limit more rapidly than a sigmoid.
The next step in that direction would be
$$
\frac1{1+\exp\left(-x-\frac{x^3}{12}-\frac{x^5}{80}\right)}=\frac12+\frac x4+O\left(x^7\right)\;.
$$
And, since you wanted to have an $x^7$ term for rapid convergence, here's the next step, too:
$$
\frac1{1+\exp\left(-x-\frac{x^3}{12}-\frac{x^5}{80}-\frac{x^7}{448}\right)}=\frac12+\frac x4+O\left(x^9\right)\;.
$$
At this stage, the plot looks very linear indeed – if you go any further, you might as well use a linear function with cut-offs at both ends instead :-)