It's helpful to consider the half-space $X=[0,\infty)\times \mathbb{R}^{d-1}$, which is the local model for a $d$-dimensional manifold with boundary. In that case one typically defines a function $f:X\rightarrow \mathbb{R}$ to be smooth, if there is a smooth function $\tilde f:\mathbb{R}^d\rightarrow \mathbb{R}$ with $\tilde f\vert_X=f.$ One can then define $\nabla f(x) = \nabla \tilde f(x)$ and you can check that for $x\in \partial X$ this is independent of the choice of extension $\tilde f$.
Requiring $f$ to have a smooth extension across $\partial X$ may feel a bit like cheating. But this is in fact equivalent to $f$ being smooth in $(0,\infty)\times \mathbb{R}^{d-1}$ (which is an open set) and all derivatives of $f$ having continuous limits as one approaches $\partial X$. See e.g. here.