Why does feature scaling by a constant vs. by a function produce different results in linear regression?

Question

If I have some data points $(x_1, x_2,\dots,x_n,y)$ where $y$ is the dependent variable and want to perform linear regression. If I perform regression with $x_1, x_2,\dots,x_n$ and $y$ vs. scaling one or multiple $x_i$ feature by some constant (say set $\hat x_i = x_i/c$), the sum of square errors of the two curves I get are identical, and the predicted $y$ value for some input are also the same for both (provided for some $x_i$ in the first model I divide that by $c$ before using it in the second model).

This isn't true if I apply a monotonic function (say logarithmic, $\hat x_i = log x_i$))? Why is this the case? (assuming for this case all the x's are positive).

Answer 1

Linear regression assumes that the dependent variable is a linear function of independent variables. Consider $x = (x_1, x_2, ...x_n)$ as input variables and $y$ as the target variable. LR finds the weights $(w_0, w_1,..w_n)$ in such a way to reduce the mean square error $MSE = \sum_{i=1}^{N} {(y_i-\hat{y_i})}^2 $ where \begin{equation} \hat{y_i} = f(x_i) = w_0 + w_1x_1+w_2x_2+..+w_nx_n \end{equation} When each of the features are scaled by some value say $c_i$, then \begin{equation} \begin{split} \hat{y_i} = f(x_i) &= w_0 + w_1x_1+w_2x_2+..+w_nx_n \\ &= w_0 + \frac{w_1}{c_1} (c_1x_1) + ..+ \frac{w_n}{c_n} (c_nx_n) \end{split} \end{equation}

i.e. when the respective features are scaled by $c_i$, LR scales the weights so as to get the same target variable. Consider the case where logarithm is applied to each feature, then

\begin{equation} \begin{split} \hat{y_i} = f(x_i) &= a_0 + a_1\log(x_1)+a_2\log(x_2)+..+a_n\log(x_n) \\ &= a_0 + log(x_1^{a_1})+..+log(x_n^{a_n}) \\ &= a_0 + log \prod_{i=1}^{n} x_i^{a_i} \hspace{2ex}\text{(non-linear)}\\ \end{split} \end{equation}

i.e. when you apply logarithm, each feature is compressed/scaled, which depends on the value being scaled i.e. $x_i$. This scale(which is not constant) value can't be captured by the LR. When the new input $k = (k_1, k_2,..k_n)$ comes at the inference time, and you apply non-linear function like logarithm, the features will be scaled differently again, than from the dataset. The bottomline is that logarithm introduces non-linearity into the features and LR can't capture this information.