let $f:\mathbb{R}^n \to \mathbb{R}^n$ be a function differentiable at a single point $p \in \mathbb{R}^n$ yet not differentiable at any other point in $\mathbb{R}^n$.
The inverse function theorem tells us that if the jacobian of $f$ is non singular at $p$ then $f$ differentiable on some open set that contains $p$.
So this means the jacobian of $f$ in our case must be singular.
Is that right?
The inverse function theorem tells us that if the jacobian of $f$ is non singular at $p$ then $f$ differentiable on some open set that contains $p$.
The differentiability of $f$ in a neighbourhood of $p$ is part of the premises of the inverse function theorem, not a conclusion of the invertibility of the derivative in one point.
Indeed, a function can be differentiable in a single point and have nonsingular Jacobian at that point.
For $n = 1$, consider a continuous nowhere differentiable function $g\colon \mathbb{R}\to \mathbb{R}$ like the Weierstraß function. Then for any $t\in \mathbb{R}$ the function
$$f(x) = (x-t)\cdot g(x)$$
is differentiable in $x = t$, with derivative $f'(t) = g(t)$, and nowhere else. If $g(t) \neq 0$, the derivative is nonsingular.
For general $n$, choose points $t_1,\dotsc,t_n$ with $g(t_i) \neq 0$ and let
$$f(x) = ((x_i-t_i)\cdot g(x_i)).$$
Then $f$ is differentiable in the point $(t_1,\dotsc,t_n)$ and nowhere else, and its Jacobian is nonsingular.
