0

I know that these "averages" questions have been asked a lot already, however I still a not sure which would be better in my situation.

I have a device that needs calibration, and gathers data over some time. In the end, the data used for calibration will be the interpolation of all the gathered data. To do so, I simply wanted to make an average of all the data: However I'm not sure which strategy would be more adapted to the situation:

Doing an average of all the gathered data at the end

Or: Making an average of the last data and the next one, and then keep the result as the last data.

The second method would obviously save some space, but in terms of outcome, I'm honestly not too sure where the difference lies.

Let's say I have a car driving the same race a hundred times. What trajectory would I get by using the first method, what trajectory would I get by using the second one? This has nothing to do with my problem, but I think I might understand better which method to use if I knew the answer to this. Let's clarify this example first, though. Imagine the trajectory of my car was plotted by an array of (x;y) coordinates. After having let my car run 4 times, I will thus obtain 4 different sets of coordinates. I now want to interpolate these coordinates to make the best possible estimate on an average set of coordinates representing the trajectory the car would take if the deviation from its path would be 0. What result would both of the methods above produce? (Namely: 1. For each coordinate in all of the sets, create a new coordinate that is (x-average; y-average) 2. For the first two coordinates, create a coordinate A((x0+x1)/2;(y0+y1)/2), then a coordinate B((xA+x2)/2;(yA+y2)/2), etc.

Thanks!

user2065501
  • 3
  • Your question could be clearer. Perhaps you can give a specific example (say you take your car example and you only race $4$ times). It also matters how much data each experiment produced, if the experiments always produce the same amount of data, if you care about behavior over time, etc. – Michael Burr Apr 16 '17 at 21:18
  • @MichaelBurr I tried clarifying the example. Is it better now? – user2065501 Apr 16 '17 at 21:38
  • @MichaelBurr I could also just put my actual problem in there, but I fear that the question won't be only a mathematical one then and thus won't fit in here anymore. – user2065501 Apr 16 '17 at 21:39

1 Answers1

2

It's hard to understand exactly what you're asking here. I'm getting that the first alternative is to average all the data, and that's easy enough to understand. So you have data $x_1\to x_n$ and at the end you summarize them by $(x_1+\ldots +x_n)/n.$

Best I can tell the second alternative is to keep a running number $x^*_n.$ When a new data point $x_{n+1}$ comes in you replace the running number with the average of it and the new data point $x^*_{n+1} = (x^*_n+x_{n+1})/2.$

This second option is, in general, a bad idea. However, it does have the advantage you mentioned of data compression. It has another possible advantage too: it is heavily influenced by the newest data point (that's half the contribution). So in the event that your old data is less indicative of the present state this can be an advantage.

However, relying this much on the latest data point is probably a bad idea in most applications. Because the data has variance, the average of the data is probably a better representation than the newest value.

Fortunately there's an easy way to get some of the advantages of the second approach without being so biased toward present data, and that's to do what you're doing with a little tweak.

Let $\lambda$ be a parameter between zero and one. You can do the same thing you want to do and keep a running value $x_n^*$ and then when you get a new value $x_{n+1}$, update it to $$ x^*_{n+1} = \lambda x^*_n + (1-\lambda)x_{n+1}.$$ So the previous method was just this with $\lambda = 1/2.$ However if you make $\lambda$ larger, say $\lambda = .9,$ you will be giving more weight to the running value and less weight to the new value. This will give you a running average where more recent values are weighted higher, but it won't be as extreme as with $\lambda = 1/2.$

This method can be better than the pure average when the data's behavior changes slowly over time so you don't want to include that that's too old. $\lambda = .94$ is for some reason the standard value most people use (I think based on a study done one time at JPMorgan on a certain data set many years ago... great reason, I know) but you can also select a value that performs best for your purposes from a backtest.

The method is usually called an exponentially weighted moving average.

spaceisdarkgreen
  • 58,508