I'm not sure if this is satisfactory, because this answer does not attempt in the slightest to generalize the notion of "differentiation", which is a very iffy matter, but there is a class of "spaces" fitting your main requirement. The fundamental insight here is that the smooth structure on a smooth manifold is determined by which functions we declare as smooth (and, so, instead of generalizing any notion of smoothness directly, we simply abstract the notion of a distinguished class of functions).
The type of "space" we need to look at is a locally $\mathbb{R}$-ringed space
, which is a tuple $(X,\mathcal{O}_X)$, where $X$ is a topological space and $\mathcal{O}_X$ is a sheaf of $\mathbb{R}$-algebras on $X$, whose stalks are local $\mathbb{R}$-algebras (meaning they are local rings whose residue fields are trivial $\mathbb{R}$-algebras). Motivating the concept of a locally $\mathbb{R}$-ringed space is beyond the scope of this post, but, roughly speaking, the sheaf should be thought of as a distinguished class of "functions" on $X$, which encode some geometric structure, and the locality conditions effectively ensure the points of $X$ can be recovered using $\mathbb{R}$-valued functions. If $(X,\mathcal{O}_X)$ and $(Y,\mathcal{O}_Y)$ are two locally $\mathbb{R}$-ringed spaces, a morphism $(f,f^{\#})\colon(X,\mathcal{O}_X)\rightarrow(Y,\mathcal{O}_Y)$ is a pair consisting of a continuous map $f\colon X\rightarrow Y$ and a morphism of sheafs of $\mathbb{R}$-algebras $f^{\#}\colon\mathcal{O}_Y\rightarrow f_{\ast}\mathcal{O}_X$ (note the contravariance). In certain geometric situations like the following, the locality will ensure that this morphism of sheafs is a literal pullback along $f$. This makes locally $\mathbb{R}$-ringed spaces into a category.
Now, a topological manifold is a topological space that is hausdorff, second-countable and locally euclidean, meaning each point has a neighborhood homeomorphic to an open subset of euclidean space. Analogously, one can show that smooth manifold is equivalent to the data of a locally $\mathbb{R}$-ringed space $(M,\mathcal{O}_M)$, such that $M$ is hausdorff, second-countable, and $(M,\mathcal{O}_M)$ is locally euclidean, meaning each $p\in M$ has an open neighborhood $p\in U\subseteq M$, s.t. $(U,\mathcal{O}_M\vert_U)$ is isomorphic as locally $\mathbb{R}$-ringed space to $(V,\mathcal{C}^{\infty}_V)$, where $V$ is an open subset of euclidean space $\mathcal{C}^{\infty}_V$ is the sheaf of smooth functions on $V$. Precisely, if $M$ is a smooth manifold, then $(M,\mathcal{C}^{\infty}_M)$ ($\mathcal{C}^{\infty}_M$ being the sheaf of smooth functions on $M$) is a locally $\mathbb{R}$-ringed space, and, conversely, if $(M,\mathcal{O}_M)$ is a locally ringed space satisfying the previous conditions, then $M$ is a topological manifold and there is a unique smooth structure on $M$ such that $\mathcal{O}_M$ becomes the sheaf of smooth functions on $M$ with respect to this smooth structure. In fact, with a bit more work, one should be able to show that this exhibits the category of smooth manifolds and smooth maps as isomorphic to a full subcategory of the category of locally $\mathbb{R}$-ringed spaces.
By the way, if you replace $\mathbb{R}$ with $\mathbb{C}$ and take $(V,\mathcal{O}_V)$, where $V$ now is an open subset of $\mathbb{C}^n$ and $\mathcal{O}_V$ the sheaf of holomorphic functions on $V$, instead, then the exact same formalism recovers the notion of a complex manifold (the designated sheaf then being the sheaf of holomorphic functions). On an even larger tangent, one also defines schemes, which are important in algebraic geometry, using this formalism. This goes to show that even thought somewhat unwieldy (and I certainly would not recommend this perspective to understand smooth manifolds), locally ringed spaces are a powerful piece of geometric formalism.
For a somewhat different and simpler formalism, you can also define smooth manifolds similarly in terms of differential spaces. I recommend reading chapter 1 of Kreck's Differential Algebraic Topology for this.
