3

I'm working with unpredictable data, it could have strong outliers in some cases, and not in others. Considering it's completely situational and random, would it make sense to just average the mean and median of the dataset to get the best of both worlds? Finding the central tendency of both common central tendencies. Or would that skewing be undesirable.

Thanks.

srb633
  • 135
  • 2
    What makes you think you would not get the worst of both worlds: sensitivity to outliers and an inability to combine with other data? – Henry May 29 '20 at 12:42
  • My inexperience with statistical science I suppose. – srb633 May 29 '20 at 13:14
  • For some distributions one can prove that the mean is 'best' according to specific criteria, for others one can probe the the median is 'best'. So you should take the population distribution into account if you know it. // But for exploring samples from unknown distributions a 'trimmed mean' is sometimes used. (1) Data are sorted from low to high. (2) A certain proportion of the observations is deleted from the very lowest and the same proportion from the highest. (3) Average what's left. Five % is often used. But as you increase the % towards 50% the trimmed mean almost becomes the median. – BruceET May 30 '20 at 08:03

1 Answers1

3

Examples of Trimmed Means

Laplace distribution- Demonstration of the trimmed mean using R statistical software. For the demo 100 observations from the heavy-tailed Laplace distribution (also known as double exponential) are used.

set.seed(530)
x = rexp(100, .1) - rexp(100, .1)
summary(x)
     Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
-35.18354  -7.33180   0.07589   0.29232   6.05352  38.23397

mean(x, trim=.05)
[1] -0.1283186
mean(x, trim=.1)
[1] -0.2133373
mean(x, trim=.25)
[1] 0.01821063
mean(x, trim=.4)
[1] 0.245436
mean(x, trim=.5)
[1] 0.07589026    # median


hist(x, br=15, prob=T, col="skyblue2",
     main="Laplace Dist'n: Sample of 100")
  rug(x)

enter image description here

Some people feel that trimmed means are better than either the mean or the median. So your idea of going somewhere between the two measurements is not bad; it's just that trimming works better than averaging mean and median.

Cauchy distribution. Another distribution, notorious for its heavy tails is the Cauchy distribution. One member of this family is Student's t distribution with one degree of freedom. The tails are so heavy that the population mean does not exist. It has been shown that an optimal way to estimate the center of a Cauchy distribution with median $\eta$ is to use a 38% trimmed mean.

Here is a random sample from a Cauchy distribution with $\eta = 0.$

set.seed(1776)
y = rt(100,1)
summary(y)
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
-47.4902  -1.2512   0.1277  -0.4320   1.1187 32.9790 
mean(y, trim=.38)
[1] 0.09840191

hist(y, prob=T, ylim=c(0,.32), col="skyblue2",
    main="Cauchy Dist'n: PDF and Sample of 100")
  rug(y)  # tick marks showing exact values
  curve(dt(x,1), add=T, col="red", n=10001)

enter image description here

Here is a brief simulation showing that a 38% trimmed mean gets a little closer to the true center of a Cauchy distribution than the mean and a couple of other trimmed means. The simulation uses $100\,000$ samples of size $n=100.$ A more extensive simulation would be be required to make a convincing case that something near 38% trimming really is best. [An analytic argument gives a result (not in closed form and requiring computational evaluation) can be used to establish the 38% result.]

set.seed(1234)
m = 10^5;  a = a.2 = a.38 = a.45 = numeric(m)
for(i in 1:m) {
 x = rt(100, 1)
 a[i] = mean(x);  a.2[i] = mean(x, trim=.2)
 a.38[i] = mean(x, trim=.38)
 a.45[i] = mean(x, trim=.45)  }
median(a); median(a.2); median(a.38); median(a.45)
[1] -0.0008814712
[1] 0.0002514082
[1] -0.0001859747  # 38% trimming
[1] -0.0003297797
Community
  • 1
BruceET
  • 51,500