0

From this resource, the writer starts with a linear model:

$$ y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + ... + \beta_m x_m $$

and then makes the RHS sigmoidal. This must then make the LHS sigmoidal to preserve the equality.

$$\implies S(y) = S(\beta_0 + \beta_1 x_1 + \beta_2 x_2 + ... + \beta_m x_m) = \frac{1}{1+ \exp^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2 + ... +\beta_m x_m)}} $$

But he instead writes

$$ P(y=1) = S(\beta_0 + \beta_1 x_1 + \beta_2 x_2 + ... + \beta_m x_m) $$

Could someone explain the implication that $S(y) = P(y=1)$?

sangstar
  • 1,947
  • I don't think it should be thought of as applying an operation to both sides and having it result in a probability. Instead, I think it's just a different choice of model which, with output in $(0,1)$, can be used to model probabilities. I think the author is providing motivation, not a derivation. See if this source is more clear: https://www.stat.cmu.edu/~cshalizi/uADA/12/lectures/ch12.pdf – Joe Jun 03 '21 at 23:43
  • @Joe Is there no other manner of mysticism justifying why logistic regression deals in probabilities other than the fact that it can purely because its range is confined to the range with which probabilities are measured? – sangstar Jun 04 '21 at 11:07
  • I believe that it is just a modeling assumption, but that it has other nice properties, such as when using MLE on a sample of data that are independent observations, the likelihood function factors nicely because it's a product of exponentials, and results in the cross entropy loss function which is convex. But I haven't seen that derivation in a while, so I'm not sure about that. Informally, I believe it is considered somewhat consistent to model probabilities with logistic regression, but I don't think there's a rigorous justification for it. Probably search/ask on cross validated for more. – Joe Jun 04 '21 at 11:26
  • Here's a related post: https://stats.stackexchange.com/questions/71176/intuition-behind-logistic-regression – Joe Jun 04 '21 at 11:35

0 Answers0