tl;dr: I've tried to construct a different way to formalize "topological spaces" than via open sets or neighborhoods. I have not seen this approach but it may have been done before. The definition as it currently stands is not satisfactory (see discussion below).
t-topologies
Consider a pair $(X,t)$ where $t:X\to \mathcal P(\mathcal P(X))$, i.e. $t(x)\subseteq \mathcal P (X)$ (which intuitively gives for every $x$ the set of subsets of $X$ that "x touches") satisfies the following axiom:
Axiom 1. For any $x\in X$ and any $U\subseteq S\subseteq X$, if $U\in t(x)$ then $S\in t(x)$. (intuitively, if $x$ "touches" the set $U$, then it also touches the same set with arbitrary additions to it).
Axiom 2. For any $x\in X$ and any $U,V\subseteq X$, if $U,V\notin t(x)$ then $U\cup V\notin t(x)$ (easily extended to infinite unions). (intuitively, if $x$ doesn't touch either of the sets $U,V$, then it also doesn't touch the combined set).
Call this pair $(X,t)$ a "touch-topology" or "t-topology" (because they're defined via "touch-relations" rather than open sets). Such a pair induces a topology: $(X,T)$, where $U\in T$ iff $\forall x\in U, U^c\notin t(x)$. (Intuitively, a set is open if none of its members touch anything outside the set).
Indeed this is a topology:
the empty set obviously is in $T$.
for any $U_i\in T$, we know that for all $x\in U_i$, $U_i^c\notin t(x)$, therefore since $(\bigcup _j U_j)^c\subseteq U_i^c$ we have $(\bigcup _j U_j)^c \notin t(x)$ by axiom 1. (intuitively, if $x$ touches nothing outside $U_i$, then it also doesn't touch anything outside $\bigcup_j U_j$).
for any $U,V\in T$, we know that for any $x\in U\cap V$ we have $U^c\notin t(x)$ and $V^c \notin t(x)$. Hence by axiom 2, $U^c\cup V^c = (U \cap V)^c\notin t(x)$. Hence $U \cap V\in T$. (easily extended to infinite intersections).
Definition. Let $\mathcal X=(X,t_X)$, $\mathcal Y=(Y,t_Y)$ be t-topologies. A function $f:X\to Y$ is touch-continuous at $x$ if for all $U\in t_X(x)$ it holds that $f(U) \in t_Y(f(y))$. (intuitively, a function is touch-continuous if it doesn't change the things that a point touches).
Theorem. Let $\mathcal X= (X,t_X), \mathcal Y = (Y,t_Y)$ be touch-topologies. If a function $f:X \to Y$ is t-continuous at $x$ then it is also "classically" continuous at $x$ in the topologies $(X,T_X), (Y,T_Y)$ induced by $\mathcal X$ and $\mathcal Y$.
proof. Assume $U\in T_Y$ (an open set). We must show that $f^{-1}(U)\in T_X$. Since $U\in T_Y$, we know that for all $y\in U$, $U^c\notin t_Y(y)$. Hence by t-continuity, $f^{-1}(U^c)\notin t_X(x)$ for all $x\in f^{-1}(U)$. Hence $f^{-1}(U)\in T_X$.
I don't think we can show that continuity implies t-continuity, though I'm not sure.
What's missing from this approach. My intention with this was to represent a topology according to the sets that an object "touches", rather than according to "open sets" (i.e. sets whose elements don't touch the outside).
This approach is similar to the "neighborhood" definition of topology.
However, my approach is not complete, in the sense that we cannot turn any arbitrary topology into a unique t-topology. That is, for any topology, there are multiple t-topologies (non-isomorphic) that induce that topology: for example, consider the topology induced by a directed graph $(X,R)$ ($R$ is a relation on $X$ denoting the arrows of the graph), where the t-topology is: given a node $x$, $U\in t_X(x)$ iff $U$ contains at least one node $y$ such that $Rxy$. It is not hard to show that the t-topology of the transitive closure of any graph induces the same topology as the t-topology of the graph itself.
My questions are:
Has this approach been done before?
Can we add a third axiom so that there is a bijection between topologies and t-topologies? And so that t-continuity and "classical" continuity are equivalent?
are there any problems with this approach?