In the optimization,i believe lots of people heard about Penalty method & Augmented Lagrangian method,but i wonder why are the creators use "penalty" and "Augmented" to name these method .
Penalty means like punishment,but i can't understand the penalty parameter punish who or which function.
Augment means increase or add,but i can't really understand what does Augmented Lagrangian method Augment,i mean ,Augmented Lagrangian method should increase or add something to let itself be better than Lagrangian method,but what is that something?
The reason i ask this is because i am afraid of misunderstanding the meaning of Augmented Lagrangian method and Penalty method,because honestly,optimization is abstract to me.
By the way , i saw this question: After reading the quadratic penalty method.i still don't understand what does it actually do,and the time of using it
Someone said the problem is easier to solve for small λ . As λ gets large the solution will lie in a steeper and steeper valley (see your picture above) which makes convergence difficult
However,wiki said The penalty method solves this problem, then at the next iteration it re-solves the problem using a larger value of ${\displaystyle \mu _{k}}$,so if the smaller ${\displaystyle \mu _{k}}$ can let the question easier to solve,why should i use bigger ${\displaystyle \mu _{k}}$ in the next iteration?
