I'm writing an assignment right now and getting into some quite lengthy derivations, which caused me to wonder about bracket notation in math and what is best practice. There are three primary types of brackets I'm familiar with: the curly brackets {}, square brackets [], and the classic parentheses (). From what I learned, in an equation with tons of brackets, you go from outer to inner in that order e.g.
$$A\left\{\frac{2}{3}\left[B^2\left(C+4\right) + 2\right] - D^2\right\}$$
My question I suppose would be what happens if you need to add another level? i.e. What is the next bracket you would use? Or would needing to do that be considered bad notation and an indication that you should simplify/expand your expression?
What is accepted practice in literature? Is there a convention, and if so what is it? Do some people not even use different types of brackets? I could see a case being made that you just stick to the round brackets () for each level, just use different sizes maybe. I had trouble finding good and insightful answers so here I am on SO. I look forward to your answers.