2

I am trying to find outliers using interquartile range and is having a few problems. I have read this article to understand how to find outliers and understood most of it.

Now I am trying to apply this method in a program. But it seems that the data I am using doesn't work with this method. The data I am using is more than 4000 rows of data and can be found in this link.

The minimum value is: 951,723112057644

The maximum value is: 1588,93458298046

Q1 Median is: 1273,39127623714

Q3 Median is: 1273,52543277336

IQR is: 0,13415653622

With the above result my inner fence range is between 1273,19004143281018 and 1273,726667577769002. Then it means that I have a lot of data are candidates to removed as outliers. I would call them outliers. The data set that I use have a lot of data between 1273 and 1274, I believe it's more than 50% that are between this range.

Is using IQR a suitable method to find outliers, are there any other method that I should use instead?

Wai Wong
  • 121
  • To me only one point looks anything like an outlier: the $1588.9$ value at the top (the second highest is $1479.1$). Otherwise you have almost $65%$ of the values between $1273.3$ and $1273.6$, and the rest spread out from near $950$ to $1490$ (hence the interquartile range of $0.134$ and the much larger standard deviation of $58.6$. This is a very highly peaked non-normal distribution but accepting that, almost all the points look plausibly coming from such a distribution. Using a $1.5$IQR rule would throw away over $30%$ of the data - not a good idea. – Henry Mar 31 '21 at 13:51
  • As you have seen the 1.5IQR rule for 'determining' outliers can sometimes give quirky results. Unless you know that a data value is faulty [documentable data entry error or eqpt failure, or obviously impossible (903 year old patient, negative weight, etc.)], you should not remove the observation from a dataset. // Boxplot outliers are characteristic of many common distributions, especially in large datasets. – BruceET Mar 31 '21 at 17:54
  • While using human eye it easy to determine that I cannot use IQR to determine outliers in this case. Is there a way to determine when to use IQR or not by code? I have more data to find outliers. What I am trying to achieve is to create a program that do this automatically – Wai Wong Apr 01 '21 at 07:42

1 Answers1

1

Continuing from the last sentence of my comment, that boxplot outliers are characteristic of many commonly occurring distributions:

Consider boxplots of 20 samples of size 100 from $\mathsf{Norm}(100,15).$

According to the 1.5IQR boxplot outlier criterion, about half of the samples show at least one outlier. While it is true that 'almost all' observations in a normal sample lie within $\mu\pm 3\sigma,$ normal tails can go to $\pm \infty.$ [Simulation using R.]

enter image description here

set.seed(2021)
m = 20; n = 100
x = rnorm(m*n, 100, 15)
g = rep(1:m, each=n)
hdr="Boxplots of 20 Normal Samples of Size 100"
boxplot(x ~ g, col="skyblue2", pch=20, main = hdr)

Among samples of size 100 from an exponential distribution with $\mu = 100$ (which is highly skewed), almost all of them have outliers. All 20 of my simulated sample have outliers.

enter image description here

R code is similar to that above; not shown in full. The key change is to use

x = rexp(m*n, .01).

BruceET
  • 51,500