Relationship between OLS estimates of slope coefficients of simple linear regression Y on X and X on Y

Question

Assume a model $y = \beta_0 + \beta_1x + u$. Given a sample $(x_i, y_i)_{i=1}^n$, we can find the OLS estimates of $\beta_1$, $\hat{\beta_1}$. Then suppose that we assume another model $x = \gamma_0 + \gamma_1y + \varepsilon$. Then we also can compute the OLS estimates of $\gamma_1, \hat{\gamma_1}$. And my question is whether $\frac{1}{\hat{\gamma_1}}$ is the unbiased estimator of $\beta_1$? Assuming the models satisfy Gauss - Markov assumptions, i.e.

1) $\mathbb{E}(u|x) = 0,\ \mathbb{E}(\varepsilon|y) = 0$

2) $(y_i, x_{i})_{i=1}^n$ are i.i.d.

3) $\mathbb{E}(y^4) < \infty,\ \mathbb{E}(x^4) < \infty$

4) $u, \varepsilon$ are homoskedastic

5) $u \sim \mathscr{N}(0, \sigma_u^2), \varepsilon \sim \mathscr{N}(0, \sigma_{\varepsilon}^2)$

What have I done:

$\hat{\gamma_1} = \frac{\sum(y_i - \bar{y})(x_i - \bar{x})}{\sum (y_i - \bar{y})^2} = \frac{s^2_{xy}}{s^2_{yy}}$ (where $s^2$ means sample covariance)

$\frac{1}{\hat{\gamma_1}} = \frac{s^2_{yy}}{s^2_{xy}} = \frac{\sum(y_i - \bar{y})(\beta_1x_i + u_i - \beta_1\bar{x} - \bar{u})}{s^2_{xy}} = \frac{\beta_1s^2_{xy} + s^2_{yu}}{s^2_{xy}} = \beta_1 + \frac{s^2_{yu}}{s^2_{xy}}$

And here I got stuck. I have no idea, how to calculate the expectation of the second term (or prove that it has zero, or non-zero expectation). Could you please give me any hints?

Thanks a lot in advance for any help!

Answer 1

There is some contradiction in your symbology: I am assuming that you estimated the betas through a regression $y,x$ and the gammas with a regression $x,y$.

That means that:
- in the first case, you are assuming the "error" to be in the $y$ data (while the $x$ values are precise);
- in the second, on the contrary, the "error" is on the $x$ data.

In any case the regression line is going to pass through $(x_{avg},y_{avg})$, so we are concerned only with determining the slope.

So $\beta_1$ is tied to the conditional probability that all the error be with $y$ and $1/\gamma_{1}$ that it be all with $x$.

The intermediate case in which we allow the error to be both on the $x$ and $y$ data is solved through Total Least Squares linear regression which would produce a different slope according to the presumed ratio of precision (or noise) between the two sets of data.

So a relation cannot be established, unless you state such an assumption.

-- addendum in reply to your comment --

The "physical" and "engineering" concept about regression is what I described above.
If you have, e.g., a set of measurements of (volume, weight) of a certain substance, made with instruments of which you know the precision and excluding systematic errors, and you want to determine the density of the substance then you resort to the statistics property of levelling out the errors by using multiple samples. That is to least squares method.
The standard least square method is to minimize $(\Delta y)^2=(y_n-(\beta_0+ \beta_1 x_n))^2$ which is the same to say that the "precise" value of $y$ is $\beta_0+ \beta_1 x_n$ which in turn means that $x_n$ is precise and you are minimizing the vertical distances $y_n$-line.
And that's acceptable if the expected r.m.s. error (the $\sigma$) on $x$ is much lower than that on $y$.
If it is the contrary, then you shall minimize the horizontal deviations from the line ($x,y$ regression).

In the case in which the expected r.m.s. errors are comparable, you shall minimize the distances from the line taken along a direction inclined same as the ratio of the $\sigma$'s (in between the two extreme cases above).

So, physically it is known that the vertical and horizontal approach might provide two different slopes, and mathematically you cannot concile them if not making assumptions about the said ratio.

-- addendum upon expliciting Gauss-Markov conditions --

Translated into the engineering approach described above, they read as:
- the systematic error is null;
- it is present on both $x$ and $y$, independently;
- it is distributed as a double Normal, on the two independent variables $x,y$, thus as a product of two Normals;
- the expected r.m.s. error is given and is respectively $\sigma_x , \, \sigma_y$.

So what explained above is totally justified and the actual slope shall be computed by minimizing the sum of the square errors = deviations taken along the line $\Delta y / \Delta x = \sigma_y / \sigma_x$.
Or, in other words, transform $x$ and $y$ into $$ \xi _{\,n} = {{x_{\,n} - \bar x} \over {\sigma _{\,x} }},\quad \eta _{\,n} = {{y_{\,n} - \bar y} \over {\sigma _{\,y} }} $$ and calculate the errors orthogonally to the line : Orthogonal Regression.

Now you expect to have the inversion of the slope if you invert the axes.

Answer 2

This is a biased estimator. If $\beta_1 = g(\gamma)=1/\gamma=$ and $\mathbb{E}[\hat{\gamma}] = \gamma=1/\beta_1$, then $\beta_1 = 1/\mathbb{E}[\hat{\gamma}]$. And due to the fact that $1/x$ is either convex (for $x > 0$) or concave (for $x<0$), $1/\hat{\gamma}$ will be biased upward and downward, respectively. This stems from Jensen inequality, i.e., for $\hat{\gamma}>0$ you have $$ \mathbb{E}g(\hat{\gamma}) \ge g\left( \mathbb{E}\hat{\gamma} \right), $$
and reversed inequality for $ \hat{\gamma}<0$.