You may rearrange the terms how you wish before actually adding and subtracting without changing the value (assuming you do it the right way). That's not what "go from left to right" is about.
Subtraction is not an associative operation, and it doesn't associate with addition either. That means that given an expression with subtraction (and possibly addition), inserting parentheses in different ways may change the outcome. For instance, in your example, we can do
$$
((3-2)+4)-1=4
$$
Or we could do
$$
3-(2+(4-1))=-1
$$
or any of a number of different orders.
Going left-to-right means that the top one is the correct one. You evaluate the leftmost binary operation (which is $3-2=1$) first, then the next one (which becomes $1+4=5$) and finally the rightmost one ($5-1=4$).
So left-to-right isn't about the terms, it's about the operations. You even follow this yourself since you in your calculations always put the next operation to the right of the previous one (abusing the $=$ sign in ways I don't really approve of, but that's another story).
You seem to have internalised the fact that in an expression like $3-2+4$, the $-$ in some sense "belongs" to the $2$, and only the $2$. That's not a bad thing in general, but in this specific case I think it made you confused about what the whole left-to-right thing is about. I personally prefer the "belongs to" interpretation over the "go from left to right" imperative, but that's not usually what's taught in schools, so it's a realisation most of us have to come to on our own.