3

The answer seems to be 'statistics' but I assume that's about as useful as saying a person interested in gravity should study 'physics'.

I anticipate a large number of data points to be delivered to me from a sensor. For purposes of discussion, say it's the temperature at the top of a tower, every second. That is, I'll have one sensor returning values every second. I know the possible range of the values, and I'll know when they were taken.

I want to know if a particular value is unusual. I want to know with some certainty that since time X, the values are trending up or down. I want to know if the sensor is misbehaving. For example, if our temperature sensor developed a bad ground and Murphy paid a visit, it could be the temperature would seem very stable. The average could be about right, and nope, not trending at all. Different analysis could point out that the nature of the data has changed, or that it is indeed just noise.

So of course this is statistics but is there a specific type I should care about?

EDIT - another reasonable use case is to notice cyclic behavior. It would probably be pretty normal for network traffic or memory usage to increase starting at about 8am 5 days a week. We would not be concerned by this. But what if one Monday, it didn't? What if another cycle was noticed, say, one that peaked on the 3rd Saturday of each month? Analysis of cyclic behavior says FFT to me, but after that I am lost.

(This may sound like a climate change question. It isn't. The fact is I am looking for a general solution. I am not even sure what the data will be. I'll ultimately have hundreds of sensors reporting to me... everything from memory usage on computers to the timestamps in which a security door is opened...)

Tony Ennis
  • 455
  • If you do not know enough statistics to answer this question, then I suggest you start wwith basic statistics – user2520938 Jul 02 '17 at 19:52
  • 1
    Perhaps time series. There are many books on that subject. https://en.wikipedia.org/wiki/Time_series – Michael Hardy Jul 02 '17 at 19:54
  • This called data analysis, it is what one does to earn money ... when one is not playing MSE ... If you have streaming data, have a rolling graph of the data & a text box that gives pertinent messages about the data ... use threshold values to decide what is important. – Donald Splutterwit Jul 02 '17 at 19:55
  • @user2520938 I know only enough to know that it's probably a branch of statistics, probably the 101 variety. But there could be something else, too. I had exactly one statistics class 30+ years ago. – Tony Ennis Jul 02 '17 at 19:55
  • Your question is about as useful as a person interested in gravity asking what they should study. You can't study a specific type of statistics, you have to study what statistics can tell you, and what not. There are lots of pitfalls, false positives and negatives, and you have to study it all to see what is likely in a given situation. –  Jul 02 '17 at 19:58
  • But surely, @ProfessorVector, I can study some without having to be an expert in all, in order to start getting value? So the question is what is the some. – Tony Ennis Jul 02 '17 at 21:18
  • A good answer to this question would look like "This problem can be solved using REGRESSION ANALYSIS. Here is a book with chapter 6 on REGRESSION ANALYSIS. You have to read ever chapters 1-5 first and do the exercises. You cannot skip to the end." – Daron Jul 02 '17 at 23:29
  • You might find statistical process control applies to your problem: https://en.wikipedia.org/wiki/Statistical_process_control – awkward Jul 03 '17 at 00:41
  • @Daron what's the book? – Tony Ennis Jul 03 '17 at 05:56
  • I don't think regression analysis is the answer (It might be though). I am just trying to counter everyone who says you cannot study a specfic type of stastics but gives no helpful advice whatsoever. – Daron Jul 03 '17 at 10:13
  • In any case you should pick up your favourite stats book and start reading. The first few chapters will definitely help you understand that specific area when you eventually find it -- because they help you understand every specific area! – Daron Jul 03 '17 at 10:14

0 Answers0