0

Say I have a set of numbers:

1, 3, 2, 1, 1, 20, 1, 3, 75, 86

I'd like to filter out all the small numbers, or "noise" in the set. The values 75 and 86 are so much higher than everything else (even 20) that I can drop everything but the 75 and 86.

A simple idea I had is to take the average, then remove anything that falls below the average. In this contrived example, the average is 19.3, which would mean the result contains 20, 75, 86. Not perfect, but a good starting point.

Is there something better I should be doing, or a standard function/technique to apply? I don't want to re-invent the wheel, but I have no idea what this "filtering" is called, so I'm not sure what to Google.

Edit:

@Matthew Conroy: The numbers come from OpenCV's cv::findContours(). The numbers represent the area of each contour. I'd like to filter out the extremely small areas and keep just the large objects that findContours() has detected.

The concept of "large" and "small" is not fixed. It depends on the numbers in the set. If a set is composed of 1, 2, 3, 1, 2, 2, 3, 4 then I guess all of them would be considered important, since there are no obvious outliers.

@Asaf Karagila please stop removing relevant tags.

Gerry Myerson
  • 179,216
Stéphane
  • 101
  • How is this a mathematical problem? – Asaf Karagila Mar 12 '17 at 06:30
  • I thought there may be a mathematical standard way to do this. Is this not a math problem? – Stéphane Mar 12 '17 at 06:31
  • What is "large" and "small" for you? As a mathematical problem, you just "remove the small numbers from the list", whatever small means for you. – Asaf Karagila Mar 12 '17 at 06:36
  • Please don't add irrelevant tags. – Asaf Karagila Mar 12 '17 at 06:37
  • It might help to mention the source of your data: that could have some bearing on how you would decided which numbers to eliminate. Why do you consider small numbers to be "noise"? – Matthew Conroy Mar 12 '17 at 06:39
  • 1
    If you hover over the tag, "filters", Stephane, you will see from the description that it has nothing whatever to do with the kind of filter you are asking about. I'm deleting it. Please don't put it back. Here's the description of the "filters" tag: Filters (and ultrafilters) are used in various areas of mathematics, e.g. general topology, set theory, boolean algebras, model theory. In topology they can be used to study convergence in a more general way than just convergence of sequences. – Gerry Myerson Mar 12 '17 at 08:12

1 Answers1

0

I did eventually find what I was looking for. For anyone else that comes across this question, things to google include:

  • segment clustering and segmentation
  • intervals
  • natural break optimization
  • kernel density estimation

A similar question I found on SO with some example code: https://stackoverflow.com/questions/17479944/partitioning-an-float-array-into-similar-segments-clustering

Stéphane
  • 101