It's not too difficult to do this reasonably fast. There is of course the problem that if N, M are around 10^13, you are adding up an awful lot of very large numbers. The result will not be able to fit int 64 bits, so that is something you need to handle. I'll leave that up to you.
N % i equals N - (N / i) * i, where N / i is an integer division, rounded down to the nearest integer. The simple trick that you need is that when i is large, there will be several or even many values of i where N / i gives the same value.
First we simplify: If N = 0 then f (N, M) = 0. If M ≥ N: N % i = N if i > N, N % i = 0 if i = N. Therefore f (N, M) = (M - N) * N + f (N, N - 1) if M ≥ N. If M is small we just calculate the sum in the obvious way, so now we have N > M >> 0. We have N / i ≥ 1.
Choose an integer K a bit below the square root of N: I'd choose K = floor (sqrt (N / 4)). We add separately for the case N / i ≥ K and N / i < K. N / i ≥ K <=> i ≤ N / K. So first add N % i for 1 ≤ i ≤ min (M, N / K). If M ≤ N / K we are done.
Now the cases where N / i < K: We don't do a loop for the values i, but for the distinct values k = N / i, for 1 ≤ k < K. For each k, determine the values i where N / i = k: We have N / i ≥ k <=> i ≤ N / k. On the other hand, N / i < k+1 <=> i > N / (k+1), so N / (k+1) < i ≤ N / k. We also need i ≤ M, so we add nothing if N / (k + 1) ≥ M. If N / (k + 1) < M, then we add for N / (k + 1) < i ≤ MIN (N / k, M). Let min = N / (k + 1) + 1, max = MIN (N / k, M), then we add for min ≤ i ≤ max.
The value to add each time is N % i = N - (N / i) * i = N - k*i. The sum is (max - min + 1) * ((N - kmin) + (N - kmax)) / 2.
So the trick is finding many values i where N/i is the same and all the values N % i can be added using a simple formula. We only add at most N/K + K values, about O (sqrt (N)).