4

I have the following set of numbers, in which I need to determine if there are any outliers: 11, 14, 21, 26, 29, 33, 61.

enter image description here

I graphed them in Desmos (see above), and visually, it appears to me that 61 is an outlier.

I searched online for a mathematical rule to determine whether a number is an outlier, and I found multiple references to the rule that a number is an outlier if it is greater than Q3, or less than Q1, by more than 1.5*(interquartile range).

However, I found two different methods for determining the interquartile range of a data set with an odd number of members:

  • method 1: Remove the median of data set as a whole. Determine the medians of the larger and smaller groups. Subtract the smaller median from the larger to get the interquartile range.
  • method 2: Consider the median of the dataset as a whole as belonging to both the larger and smaller groups. Determine the medians of the larger and smaller groups. Subtract the smaller median from the larger to get the interquartile range.

Method 1 gives the groups (11, 14, 21) and (29, 33, 61), which has medians 14 and 33, for an interquartile range of 19.

Method 2 gives the groups (11, 14, 21, 26) and (26, 29, 33, 61), which has medians 17.5 and 31, for an interquartile range of 13.5.

(Incidentally, the TI-83 calculator uses method 1, and Microsoft Excel uses method 2.)

Getting back to the 'determining outliers' question:

  • Using method 1 gives 33 + 1.5*19 = 61.5. Since this is greater than 61, by this method, there are no outliers in the data set.

  • Using method 2 gives 31 + 1.5*13.5 = 51.25. Since this is less than 61, by this method, 61 is an outlier.

So my question is: Is one of these methods considered more correct than the other? Or is the whole 'point' of outliers more intuitive, and less bound by actual formulas, so the right method would be just looking at my graph instead? (If possible, please include links to any authoritative source or reference that you may know of. Thank you.)

EDIT: After reading some of the responses, I think I need to add a bit more context info here. I am reviewing an Algebra textbook meant for middle to high school students, and this is part of a question which appears in that book, specifically in the area of the book where outliers are being taught about. It looks to me like the numbers in this question were specifically chosen to have one be as far away as possible from the rest, while having it not be considered an outlier by the 1.5IQR formula.

I think that this will give students the perspective that outliers are always determined by a rigid formula, without any sort of intuitive 'that point looks far away from the rest' outlook. I am looking for input on whether this a correct / proper outlook to be teaching.

Phil
  • 161
  • The reason the is such a large difference between the techniques is the small sample size. Maybe you should consider the median as "half" a sample in each of the two halves. – Myridium Jan 04 '18 at 20:40
  • Also generally, for a distribution of samples on the real number line, we would consider outliers to be those samples some number of standard deviations from the mean. – Myridium Jan 04 '18 at 20:42
  • 1
    About ten different methods for finding quantiles (such as quartiles and the median) are in regular use in various statistical software packages. For example a compromise is necessary in finding the median if the sample size is even; most rules take the average of the middle two observations. The possibility of ties can cause further complications. Differences between methods become unimportant when the sample size is large. // Methods not based on quartiles are also used to find outliers. – BruceET Jan 04 '18 at 23:24
  • 2
    (1) Particularly for beginners such as HS students, I agree that it might be useful to have a concrete standard definition of 'outlier'. If you're using a particular definition of quantile and want to use the 1.5IQR rule, I guess you can arbitrarily declare one for your class. But in general statistical practice, there is no such thing. (2) It would also be appropriate to for HS school students to know that outliers are to be "noticed" and "checked for accuracy, but never "automatically discarded." ... – BruceET Jan 08 '18 at 23:58
  • 1
    ... (3) It may be hard to deal with large datasets at the HS level, but I think it is crucial for students to understand that boxplots and any boxplot oulier rules are on questionable ground if $n < 50,$ say. Quartiles of a sample of size 7 are silly and useless. (4) There is a tendency of math teachers in gen'l and HS teachers in particular to view math as a context for "certainty" and applying "absolute rules." Stat is a math sci, but it is inductive rather than deductive. Stat analysis might be a forum for discussing the deliberative judgment so widely req'd in real life. – BruceET Jan 09 '18 at 00:09

1 Answers1

3

Using boxplots to identify outliers. Roughly speaking, a boxplot is based on the median and the lower and upper quartiles ($Q_1$ and $Q_3,$ respectively) and the median. The 'box' extends from $Q_1$ to $Q_3.$ The median is shown by a symbol within the box (usually a line). 'Whiskers' extend from $Q_1$ down to the minimum, and from $Q_3$ up to the maximum. If the boxplot shows outliers, then the they are often values more than $1.5\,\text{IQR}$ below $Q_1$ or above $Q_3,$ where IQR = $Q_3-Q_1$ is the 'interquartile range' Then whiskers stop at the most extreme observations that are not outliers.

Here is an a boxplot drawn according to the conventions for quantiles in R statistical software. According to this boxplot, the observation at 51 is an 'outlier'. There is no upper whisker for your sample because observation 33 is the upper quartile. (Among all sample sizes, perhaps $n = 7$ causes the most difficulty in making boxplots, which are at their best for moderate or large samples.)

enter image description here

Using sample means and standard deviations to identify outliers. Another method of identifying outliers requires more computation. Each suspect observation in turn is removed from the sample and then the mean $\bar X^\prime$ and SD $S^\prime$ of the remaining $n-1$ observations is found. If the temporarily deleted observation is outside the interval $\bar X^\prime \pm kS^\prime$ (for some fixed $k,$ often 2.5 or 3), then that observation is considered an outlier. Then on to the next suspect observation. With either $k = 2.5$ of $3,$ this method identifies 61 as an outlier in your sample. [If the observation itself were used in finding the interval $\bar X \pm kS,$ then it might inflate $S$ so that it could not be seen as an outlier. For your sample, $\bar X + 2.5\,S = 69.35,$ so 61 would not be identified as an outlier.]

Treatment of outliers in data analysis. Whatever the method of identification, 'outliers' are worth a second look to make sure they did not result from some accidental factor such as data-entry error or equipment failure. In data analysis, they should not be routinely disregarded without good cause.

Sometimes outliers are among the most important observations. In some emerging technologies they may represent the signal making itself known above the noise. [The few signals at CERN that led to the confirmation of the Higgs boson were outliers.] In earthquake data, the only observations of great interest to the general public are outliers. [Quakes of magnitude less than 1 are not routinely recorded; only those above about 4 are noticeable, except just above the epicenter; only those above about 5.5 may be widely felt. A 5.5-magnitude quake last night was strongly felt for about 25 miles in all directions from Berkeley, CA because it occurred very deep in its fault.]

The high probability of seeing outliers in certain circumstances. Some long-tailed distributions that occur regularly in practical applications almost always show boxplot outliers. Exponential distributions widely used in modeling the lifetimes of certain kinds of mechanical and electronic components are examples. Here are boxplots and their outliers for 20 samples each of size 50 from an exponential distribution with failure rate $\lambda = 1/5$ per unit time or mean lifetime $\mu = 5$ time units.

enter image description here

This is not an example rigged to make a point. It would be typical for about 90% of such samples to show boxplot outliers. One might say that showing high outliers is a characteristic of exponential data.

Sometimes elementary texts suggest that showing outliers is an indication that a sample did not come from a normal distribution. This is misguided advice. Often a normal sample does have all its observations within three standard deviation of its mean. However, the tails of a normal distribution extend to $\pm \infty$ and it is not rare to see boxplot outliers in truly normal samples of moderate size. The figure below shows 20 samples of 50 observations (perhaps test scores) from a normal distribution with mean 100 and standard deviation 15.

enter image description here

It is typical for about 37% of such samples to show boxplot outliers.

BruceET
  • 51,500
  • Thank you for your detailed response. I have added an edit to my original question, which I hope clarifies where I am coming from, and what sort of additional information / perspective I am looking for. – Phil Jan 08 '18 at 22:53