You write, for $f(x)=x^2$ that
When $x$ changes from $0$ to $1$, $f(x)$ changes from $0$ to $1$ and $f'(x)$ is $2$.
But that's not true! It's true that $f'(1)=2$, but you're talking about a whole interval of numbers here, you can't just evaluate $f'$ at one point and expect that it captured everything about your function. On that interval, $f'$ actually varies between $f'(0)=0$ and $f'(1)=2$.
If you want to look at a difference over an interval, you would have to take the average of the derivative over this interval which, since $f'$ happens to be linear here, is $1$ - exactly that average rate of change you observed. However, you should note that $f$ changed more quickly in the latter half of this interval than in the former - for instance $f(1/2)=1/4$, so the interval from $0$ to $1/2$ had less total change than from $1/2$ to $1$. The derivative is sensitive to what happens on small intervals and correctly recognizes that $f$ increases very slowly near $0$, but makes up for this by increasing very quickly near $1$.
Formally, saying $f'(1)=2$ means that over all small enough intervals, your velocity will be as close as you want to $2$. For instance, for quadratic functions, you can say that is $|x - 1| < \delta$ then the slope from $(1,f(1))$ to $(x,f(x))$ is within $\delta$ of $2$. This is exactly what the definition
$$f'(x)=\lim_{h\rightarrow 0}\frac{f(x+h)-f(x)}h$$
says, expanded out via the definition of a limit - and it shows that the derivative is really just the slope measured along sufficiently small intervals to avoid the issue of average velocities.