2

I am studying this paper about logistic regression. In section 4.2 (Randomly Generated Problems) on page 1534, they say "Features of positive (negative) examples are independent and identically distributed, drawn from a normal distribution $\mathcal{N}(\nu,1)$, where $\nu$ is in turn drawn from a uniform distribution on $[0,1]$ $([−1,0])$."

Question1: Why all entries of positive(negative) examples are drawn from the same distribution? More clearly, let $x\in \mathbb{R}^n$ be an example vector whose label is positive. Then all entries of $x$ are coming from a ball centered at $[\nu, \dots, \nu]^{\top}$ whose radius is roughly 1 because the variance is 1. Similarly, for an example vector whose label is negative all entries of $x$ are coming from a ball centered at $[-\nu, \dots, -\nu]^{\top}$ whose radius is roughly 1 because the variance is 1. Am I understand this correctly? The following is the picture of these balls when the standard deviation is 0.1. If we let the standard deviation be 1 they are overlapped. enter image description here

My thoughts: With the above set up we have two balls one in non-negative orthant and the other in non-positive orthant. Clearly, we can find a hyperplane that can separate them. This makes finding the separating hyperplane easy. We can have our two balls of data at any point of $\mathbb{R}^n$. To do that I have the following suggestion.

Question 2: Isn’t it better to sample each $i$-th entry of example vectors with positive label from $\mathcal{N}(\alpha_i,1)$ and $i$-th entry of example vectors with negative label from $\mathcal{N}(\beta_i,1)$? Then the center of example vectors with positive label would be $[\alpha_1,\dots,\alpha_n]^{\top}$ and the center of example vectors with negative label would be $[\beta_1,\dots,\beta_n]^{\top}$. Of course, $\alpha_i$'s and $\beta_i$'s are drawn from identically and independently from $\mathcal{N}(0,1)$. The following is one realization of what I explained. enter image description here

Saeed
  • 175

1 Answers1

1

Your picture about the balls is not correct. If $x$ is sampled from a normal distribution $N(0,1)$, then there is a substantial chance that $|x|$ will exceed 1. A more subtle point is that it sounds like $\nu$ is independently sampled for each data point.

Here is Python code to generate the examples like the describe:

import numpy as np
import matplotlib.pyplot as plt
N=200
Xpos=np.random.rand(N,2)+np.random.randn(N,2)
Xneg=-np.random.rand(N,2)+np.random.randn(N,2)
plt.scatter(*Xpos.T)
plt.scatter(*Xneg.T)`

and the result:

enter image description here

Simon Segert
  • 5,487
  • You are right about my pictures because after finding the center I used $\sigma=0.1$. About your point "A more subtle point is that it sounds like $\nu$ is independently sampled for each data point.", even if $\nu$ is independently drawn for each entry, still positive samples have mean greater than zero and negative ones have mean of negative which make our data biased. I am right? – Saeed Jan 11 '22 at 02:00
  • sure, positive examples will be biased towards positive means, and negative examples towards negative means. but i'm guessing this is by design; if there were no such bias, then there wouldn't be any signal that the model could pick up on to differentiate the classes. – Simon Segert Jan 11 '22 at 15:55
  • Unfortunately, the way this paper generates the synthetic data, there is such a bias. That is why I suggested to pick the means from a random normal distribution and then perturb them by another random normal. If we do that your code needs an n to change rand to randn. I think in this way there would not be any bias. Correct me if I am wrong. – Saeed Jan 11 '22 at 21:49
  • you're right that would eliminate any difference between the distributions. but why would you want to do that? if both the positive and negative samples are drawn from exactly the same distribution, then the labels are basically just random, and there is no structure that the regression model could possibly detect. – Simon Segert Jan 12 '22 at 04:13
  • I am trying to generate reliable data for an optimization problem that is not straight forward to solve. If I pick data with some bias, then any hyperplane passing from the origin would be the answer. I want to make it harder. – Saeed Jan 13 '22 at 01:11