Most mathematical theories for music consider the issue of consonance/dissonance, but in music, we actually care more about the stability of notes in a scale. For example, the subdominant is unstable and tends to resolve to the mediant. This has nothing to do with dissonance, but only how close something is to the tonic triad.
Question: is there any mathematical theories that discuss such instability of subdominant?
Well, all these rules that you are referring to have to do with Classical harmony. I mean, trying to write modern atonic music or automatised music or even traditional/folk music makes all these rule almost useless (consider Xenakis or traditional music from Caucassus/Pontos where it is usual to start on the tonic and end with the subdominant (!)).
However, let us stick, for reasons of simplicity to classical music theory. According to this, one may consider all the "traditional" rules of harmony (the subdominant resolving to the dominant with non-parallel voice-movement, ending on the tonic either of the intial tonality or that of the homonymous major etc) as sentences or types in a First Order Language and take it as a ground Knowledge Base.
Starting with this, one can reason based on this set of rules and get some "elaborated" auto-tune tool which not only "rounds" towards the closest note (in terms of absolute wave-frequency) but also alters the given melody so as to adhere to some of the rules of Classical-harmony.
Nevertheless, bare in mind that such a formalization as briefly described above bears little resemblance to any algebraic theory analysing e.g. counterpoint.
I hoped this helped, at least a little!
