0

I have a time serie with 1000 values, but some values are out of logic. Let's say I have points like that:

[[...]100,101,100,99,97,103, 10,5,2, 100,101,102,97[...]]

(there are much more value in the time series) But how to remove this "noise" (10,5,2) ? Is there any mathematical way to do that ?

Thanks

Unitech
  • 105
  • 1
    The word "outlier" is often used to refer to data points that depart substantially from the distribution of fit of the majority of data points, but I'm not familiar with the term "unlogical" or phrase "out of logic" in data sets, specifically time series. – hardmath Aug 06 '14 at 19:06
  • How do you know it is noise? – copper.hat Aug 06 '14 at 19:10
  • Nice tips for the "outlier" thing, thank you hardmath. I found this module to help me to find this: https://github.com/pablodenadai/outlier/blob/master/lib/outlier.js. I know that is noise because its meaning less point – Unitech Aug 06 '14 at 19:11

1 Answers1

0

You can use a low pass filter to cancel high frequency terms. Low-pass filters provide a smoother form of a signal, removing the short-term fluctuations, and leaving the longer-term trend. For the theory and reference, check wiki link

It helped me to cancel some noises in my program via the code below.(You may need to modify in your program language)

$ y[0] := x[0]$

for $i$ from $1$ to $n$

 y[i] := α * x[i] + (1-α) * y[i-1]

return y

where $ x[i] $ is your input series,

$ y[i] $ is your output series ,

$α$ used for cut-off frequency , select a real number between $0$ and $1$

But please note that, this code can change your other terms little bit too because of filter cutt-off frequency or non-ideal filter characteristics . You must find appropriate cutt of frequency and maybe you can need to use higher degree filter to get ideal filter behaviour.

You can also check that to see how to design a digital filter

Mathlover
  • 10,058
  • Thanks for you great answer. So should I use "outlier" stuff (https://github.com/pablodenadai/outlier) or low pass filter (https://github.com/LukaszKrawczyk/LPF) ? – Unitech Aug 06 '14 at 20:03
  • @tknew the second link shows in examples how it works . If It looks ok to use for your purpose , you may try the library. I think first one finds only one strange value but you need to define more. Maybe you can run three times the function to find your three strange value via using first link library. Really this depends on you. Mostly filters have been used for signals that flows in real time. This kind of real time signals can have any time noise and we cannot estimate how many input affected. – Mathlover Aug 06 '14 at 20:24