1

I have a single regressor $X$ and response $y$, where $y=n_k/n$ if $X$ is of class $k$, and $k=1,2$. Let $n_k$ denote the number of observations in class $k$, and $n$ the total number of observations. Denote $\mu_k$ as the mean of class $k$, $\Sigma$ the covariance matrix of X, and $\pi_k$ the probability of class $k$ occurring. I have the general discriminant of $\delta(x)=x'\Sigma^{-1}\mu_k-\frac{1}{2}\mu_k'\Sigma^{-1}\mu_k+\log(\pi_k)$,

and I know the decision boundary between classes 1 and 2 is given by $$\log(n_2/n)-\frac{1}{2}\mu_2'\Sigma^{-1}\mu_2+x'\Sigma^{-1}\mu_2=\log(n_1/n)-\frac{1}{2}\mu_1'\Sigma^{-1}\mu_1+x'\Sigma^{-1}\mu_1$$

I let $\hat\sigma^2$ estimate $\Sigma$ and $\hat\mu_k$ estimate $\mu_k$, and then by treating these as scalars and using algebraic manipulation, I obtain

$$\frac{(\hat\mu_2-\hat\mu_1)x}{\hat\sigma^2}=\frac{(\hat\mu^2_2-\hat\mu^2_1)}{2\hat\sigma^2}-\log(n_2/n_1)$$

For any $X$ that doesn't satisfy this equation, I suspect that determining which side of this equation is greater for a given $X$ will tell me which class $X$ belongs to, but I can't figure out which side would correspond to which class (if this is a correct assumption).

sk13
  • 49

1 Answers1

1

When we are not sure which side does a classifier assign a data point to, a possible way is to substitute some points that we know it belongs to a particular class to observe the outcome. We can try out with $\hat{\mu}_1$ and $\hat{\mu}_2$ where $\hat{\mu}_1\ne\hat{\mu}_2$

The boundary is

$$\frac{(\hat\mu_2-\hat\mu_1)x}{\hat\sigma^2}-\frac{(\hat\mu^2_2-\hat\mu^2_1)}{2\hat\sigma^2}=-\log(n_2/n_1)$$

Let $$f(x) = \frac{(\hat\mu_2-\hat\mu_1)x}{\hat\sigma^2}-\frac{(\hat\mu^2_2-\hat\mu^2_1)}{2\hat\sigma^2}=\frac{(\hat{\mu}_2-\hat{\mu}_1)}{\hat{\sigma}^2}\left( x-\frac{\hat{\mu}_1 + \hat{\mu}_2}2\right).$$

$$f(\hat{\mu}_1) =\frac{(\hat{\mu}_2-\hat{\mu}_1)}{\hat{\sigma}^2}\left( \hat{\mu}_1-\frac{\hat{\mu}_1 + \hat{\mu}_2}2\right)=\frac{-(\hat{\mu}_2 - \hat{\mu}_1)^2}{2\hat{\sigma}^2} <0$$

$$f(\hat{\mu}_2) =\frac{(\hat{\mu}_2-\hat{\mu}_1)}{\hat{\sigma}^2}\left( \hat{\mu}_2-\frac{\hat{\mu}_1 + \hat{\mu}_2}2\right)=\frac{(\hat{\mu}_2 - \hat{\mu}_1)^2}{2\hat{\sigma}^2} >0$$

Hence if $f(x) < -\log(n_2/n_1)$, it belongs to the first class.

If $f(x) > -\log(n_2/n_1)$, it belongs to the second class.

Siong Thye Goh
  • 149,520
  • 20
  • 88
  • 149