Let $f(x) \in L^1 [a, b]$ satisfies $\lim_{h \to 0} 1/h \int_{a}^b |f(x+h)-f(x)|dx = 0$. I want to prove that there exists $c$ such that $f(x) = c$ a.e.
Morally what happens is that $\int_{a}^b |f'(x)|dx = 0$ which gives the desired result but $f(x)$ is not necessarily differentiable. How can I show this rigorously?
Here are two different approaches to the problem in the OP. I leave most of the details to the asker.
Definition: Consider $f\in L^p(\mathbb{R})$. A function $g\in L_p$ is the $L_p$ derivative of $f$ if $$\lim_{h\rightarrow0}\Big\|\frac1h(f(\cdot + h)-f(\cdot))-g(\cdot)\Big\|_p=0$$.
The following is a well known result in harmonic analysis (see Stein, E. and Weiss G., Introduction to Fourier Analysis on Euclidean spaces, Princeton Universe. Press, 1975, Chapter 1)
Theorem: If $f\in L_1(\mathbb{R})$ and $g$ is its $L_1$-derivative, then $$\widehat{g}(t)=2\pi i t \widehat{f}(t)$$ where $\widehat{\phi}(t)=\mathcal{F}\phi(t)=\int_{\mathbb{R}} e^{-2\pi itx}\phi(x)\,dx$ is the Fourier transform of a function $\phi\in L_1$ In particular, if $f$ has $L_1$-derivative equal to $0$, then $\widehat{f}=0$ and so $f=0$ (a.s.)
Proof: Notice that for $f\in L_1$, if $f_h(\cdot)=f(\cdot+h)$, then $\widehat{f_h}(t)=e^{2\pi its}\widehat{f}(t)$. Since $|\widehat{\phi}(t)|\leq\|\phi\|_1$ for any $\phi\in L_1$,
$$\Big|\frac{e^{2\pi iht}-1}{h}\widehat{f}(t)-\widehat{g}(t)\Big|\leq\Big\|\frac{1}{h}\big(f(\cdot+h)-f(\cdot)\big) -g(\cdot)\Big\|_1\qquad\Box$$
Example: Consider $f(x)=\mathbb{1}_{(0,1]}$. Then, for $h>0$ $$\frac{f(x+h)-f(x)}{h} =\frac{1}{h}\big(\mathbb{1}_{(-h,0]}(x)-\mathbb{1}_{(1-h,1]}(x)\big)\xrightarrow{h\rightarrow0}0\qquad\text{a.s}$$ Similar for $h\rightarrow0-$. However, $g=0$ is not the $L_1$-derivative of $f$, for $$\Big\|\frac{1}{h}\big(f(\cdot+h)-f(\cdot)\big)\Big\|_1=2\qquad\Box$$
If in the OP we change $(a,b]$ for $\mathbb{R}$, or $f\in L_1(a,b]$ is extended extended as zero outside $(a,b]$ and the differential condition is set to $$\lim_{h\rightarrow0}\frac1h\int_\mathbb{R}\Big|\big(\mathbb{1}_{(a,b]}f\big)(x+h)- \big(\mathbb{1}_{(a,b]}f\big)(x)\Big|\,dx=0,$$ then the Theorem above implies that $f=0$ (almost surely).
Definition: For $f\in L_p(\mathbb{S}^1)$, a function $g\in L_p(\mathbb{S}^1)$ if $$\lim_{h\rightarrow0}\Big\|\frac{f(\cdot+h)-f(\cdot)}{h} -g(\cdot)\Big\|_p=0$$
A similar result to the Theorem above holds. For any $\phi\in L_1(\mathbb{S}^1)$ let $\hat{\phi}(n)=\int^1_0 e^{-2\pi inx}\phi(x)\,dx$ be its $n$-th Fourier coefficient, $n\in\mathbb{Z}$. If $f\in L_1(\mathbb{S}^1)$ admits an $L_1$-derivative $g$, then $$ \Big|\frac{e^{2\pi ihn}-1}{h}\hat{f}(n)-\widehat{g}(n)\Big|\leq\int_{\mathbb{S}^1}\Big|\frac{f(x+h)-f(x)}{h}-g(x)\Big|\,dx$$ Hence, for $n\in \mathbb{Z}$, $$2\pi in\widehat{f}(n)=\widehat{g}(n)$$ Consequently, if $g=0$, then $\widehat{f}(n)=0$ for all $n\in\mathbb{Z}\setminus\{0\}$. Therefore, $f=\widehat{f}(0)$ a.s.
I will assume that $f$ is defined outside of $[a, b]$ in a suitable way so that it is locally integrable on $\mathbb{R}$.
Define $F$ by
$$ F(x) = \int_{a}^{x} f(t) \, \mathrm{d}t. $$
Then it is well-known that the set $E$ of points $x$ in $[a, b]$ where $F'(x) = f(x)$ is Lebesgue-full in $[a, b]$, that is, $[a, b] \setminus E$ has zero Lebesgue measure. Now for any two points $c, d \in E$ with $ c< d$ and $h > 0$,
\begin{align*} &\left| \frac{F(d+h) - F(d)}{h} - \frac{F(c+h) - F(c)}{h} \right|\\ &\quad= \left| \frac{1}{h} \int_{c}^{d} (f(x+h) - f(x)) \, \mathrm{d}x \right| \leq \frac{1}{h} \int_{c}^{d} \left| f(x+h) - f(x) \right| \, \mathrm{d}x. \end{align*}
Then, letting $h \to 0$ and using the assumption, we have
$$ \left| f(d) - f(c) \right| \leq 0 $$
and hence $f(d) = f(c)$. Since this is true for any $c, d \in E$, we find that $f$ is constant on $E$, proving the claim.
We define the difference quotient of $f$ by
$$\Delta_h f(x):= \frac{f(x+h)-f(x)}{h}.$$
If we restrict ourselves to the intervall $(a+\epsilon,b-\epsilon),0<\epsilon<<1,$ and choose $0<\lvert h\rvert < \epsilon$, then $\Delta_h f(x)$ is well-defined for every $x\in (a+\epsilon, b-\epsilon)$. Now select an arbitrary test function $\phi \in C_c^{\infty}(a+\epsilon, b-\epsilon)$. Since $\phi$ is bounded, we have $$\left\lvert\int_{a+\epsilon}^{b-\epsilon} \Delta_h f(x) \phi(x)\,dx\right\rvert \leq \lvert\lvert \phi\rvert\rvert_{\infty} \int_{a+\epsilon}^{b-\epsilon} \lvert \Delta_h f(x)\rvert \to 0,\quad h\to 0$$ by assumption. If $\langle \cdot,\cdot\rangle$ denotes the inner product of $L_2$, it follows that $$0=\lim_{h\to 0}\langle \Delta_h f(x), \phi(x)\rangle = -\lim_{h\to 0}\langle f(x),\Delta_{-h} \phi(x)\rangle = - \langle f(x),\phi'(x)\rangle,$$ where we have used the "integration by parts"-formula for difference quotients and the fact that $\Delta_{-h}\phi$ converges uniformly to $\phi'(x)$ (Uniform convergence of difference quotients to the derivative). The lemma of du Bois-Reymond finally yields $f\equiv c$ on $(a+\epsilon, b-\epsilon)$ for some constant $c$. The result follows by letting $\epsilon \to 0$.
As others have mentioned, your assumption currently ist not well-defined, but the proof shows that we only need $\int_{a+\epsilon}^{b-\epsilon} \lvert \Delta_h f(x)\rvert \to 0,h\to 0$ for every $0<\epsilon <<1$.
