I'm working on a computer program that has to stabilize a set of points which should all appear on a plane in 3D space. Currently the program does not use the knowledge that all points should appear on a plane. So I want to write a program that projects these points on a plane. Now I know the math on how to project points onto a plane along a vector. Additionally rather then fitting for minimal distance I have noticed that the program tends to make errors in estimating distance but this error typically only occurs for a few data points meaning that I think it's best to function fit to the square root of the distance rather then the distance or the square of the distance (as I know is more common).
Note that while I'm programming in unity/C# I'm fine with a more or less mathematical answer (as long as the notation is not overly exotic)