This isn't well-defined unless $m$ divides $n$. To see why, consider $m=3$, $n=4$, $a=b=1$. Then you're looking to reduce $(1\pmod{4})\pmod{3}$. $1\pmod{3}$ is the set of integers which give that remainder upon division by 4: $\{\dots,-3,1,5,9,\dots\}$, and taking each of those modulo 3, we find $\{\dots,0,1,2,0,\dots\}$, that is, you don't always get the same value!
Trying instead $m=2$, we see that taking modulo 2 finds $\{\dots,1,1,1,1,\dots\}$. And, more generally, if $n=km$, then $ab\pmod{n}$ is simply the task of writing $ab=xn+r$ where $0\le r<n$, but this rewrites as $ab=xkm+(ym+s)=(xk+y)m+s$ where $0\le s<m$, so $s$ is the remainder modulo $m$ and this reduction makes sense.
In this second case, taking modulo $n$ and then modulo $m$ is a little redundant so you might as well just take modulo $m$ throughout.