We will assume that $h$ approaches $0$ from the right. A similar argument deals with the general case. We will need to make some assumptions about $f$, since the result does not hold in general. We will show that if $f$ is continuous in some interval $[t,t+b]$, then
$$\lim_{h\to 0^+} \frac{1}{h}\int_t^{t+h} f(s)\,ds=f(t).$$
We need to assume (or prove) certain properties of the definite integral. Let $m_h$ be the minimum of $f(s)$ in the interval $[t,t+h]$, and let $M_h$ be the maximum of $f(s)$ in this interval. These both exist by continuity.
From any of the standard definitions of the definite integral, we can show that
$$h\, m_h \le \int_t^{t+h} f(s)\,ds\le h\, M_h.$$
Divide through by $h$. We get
$$m_h \le \frac{1}{h} \int_t^{t+h} f(s)\,ds\le M_h.$$
Now let $h\to 0^+$. Then $m_h\to f(t)$ and $M_h\to f(t)$, and now Squeezing gives us the desired result.
Another way: Or else we can use more machinery, like the Fundamental Theorem of Calculus. Consider the function $g(h)$ given by
$$g(h)=\frac{\int_t^{t+h}f(s)\,ds}{h}. $$
Using L'Hospital's Rule and the Fundamental Theorem, we find that
$$\lim_{h\to 0}g(h)=\lim_{h\to 0}\frac{f(t+h)}{1}.$$
The limit on the right is $f(t)$.
Remark: We stayed away from using the term "infinitesimal," since in the reals there is no positive element with is $\lt \frac{1}{n}$ for every positive integer $n$. One could reframe the first argument above using the language of non-standard analysis, but there seems to be no great advantage to doing so.