0

Suppose that I have temperatures (T1) in x-axis and temperatures' frequencies (F1) in y-axis. So, that for each of the 10 temperatures in x-axis I get related frequencies in y-axis. I have also temperatures (T2) in another graph (in x-axis) and frequencies (F2) in y-axis. I am thinking of using Least mean squares regression in each graph. Can I use y1 = a1x1 + b1 (for T1) and compare it with y2 = a2x2 + b2 (for T2) in order to find a percentage (%) similarity between the 2 temperature graphs? Is there any better way (solution) ?

just_learning
  • 117
  • 1
    The question is unclear. If you have a plot of frequencies of the different temperatures, that is just a histogram. Essentially, you have univariate data, not bivariate data so can't do least squares. – Golden_Ratio Feb 14 '22 at 22:58
  • True! How can I compare similarity between T1 and T2 ? – just_learning Feb 14 '22 at 23:01
  • 2
    Please use MathJax. See https://math.meta.stackexchange.com/questions/9959/how-to-ask-a-good-question – blamethelag Feb 14 '22 at 23:02
  • 1
    @just_learning It's unclear how you want to compare the two though; are $T_1,T_2$ jointly obtained (so that $(T_1,T_2)$ is bivariate data)? If not, one possibility if you want compare their histograms is to normalize each histogram so the frequencies sum to one and treat it as a probability distribution. And then use some kind of statistical distance to measure how different the two distributions are (examples of such distances would be KL divergence, earth mover distance, etc.) – Golden_Ratio Feb 14 '22 at 23:06
  • I am studying KL divergence. Can I transform the KL divergence to percentage? Meaning 100 % ---> full similarity and any other value just lower similarity? – just_learning Feb 15 '22 at 09:59
  • @Golden_Ratio: What is the rationale between histogram normalization? I mean: which is the suitable filter to normalize my histogram? Also, If I normalize the histogram, in order to make the above mentioned comparison between T1 and T2, don't I distort the histogram? – just_learning Feb 16 '22 at 10:28
  • 1
    @just_learning By "normalize," I just meant divide each frequency by the sum total frequencies so that you have a valid probability mass function. This doesn't distort anything (you preserve the relative frequencies). Essentially you are looking at the EDF. Then you have two different probability distributions (one for T1, and one for T2) and can use KL divergence or another kind of statistical distance to compare them. – Golden_Ratio Feb 16 '22 at 14:45
  • @just_learning see my response below – Golden_Ratio Feb 16 '22 at 15:14

1 Answers1

1

So it looks like you have two sets of univariate data: temperatures $T_1$ and temperatures $T_2$. Further, you wish to find a way to compare how "close" the two datasets are to each other, where their "closeness" is measured on a unit interval $[0,1].$

One way to do this is obtain the EDF for each dataset. Then you can use a statistical distance to compare them. One distance you may use is Jensen-Shannon divergence, which is a symmetrized and smoothed version of KL divergence. The advantage of this distance is that it is between 0 and 1 (when using log base 2). You may choose to use the square root of Jensen Shannon divergence to make it a valid metric. Thus, you can roughly say that one minus the Jensen Shannon divergence (or its square root) gives a way to measure "percent similarity" between your two datasets. This distance has applications in e.g. genome comparison and machine learning. It may also help to review the literature to see applications for your kind of setting.

Golden_Ratio
  • 12,591
  • Ok, I am using Jensen-Shannon divergence, it seems to work. It gives me 100% where the 2 histograms are the same and lower than 100% when they differ. The problem is how to find a threshold, because it gives me most of the time > 89% similarity... Any idea, anyone? – just_learning Feb 17 '22 at 19:54
  • 1
    You can try using the square root of JS divergence, which is a metric. But what kind of "threshold" are you looking for? You just want a % similarity right? – Golden_Ratio Feb 17 '22 at 19:59
  • So, for SQRT(JS divergence) and up as a sure similarity and above the SQRT(JS divergence) not. Something like thiat? – just_learning Feb 17 '22 at 20:03
  • 1
    Hm, sounds like you are doing a hypothesis test then. I wonder whether Kolmogorov Smirnov would better suit your needs then: https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test – Golden_Ratio Feb 17 '22 at 20:07
  • 1
    JS divergence is just a way to measure the distance between distributions. However, I wouldn't suggest it for testing whether they are similar as there are formal tests you can use to control the type 1 error – Golden_Ratio Feb 17 '22 at 20:11
  • Thank you all! I am studying all these! One last question, if I have the same concept as T1 and T2, but for more histograms (T1, T2, T3, T4, T5, T6). How can I do the comparison? Is it ok to find theD1= JSD(T1, T2), then D2 = JSD(T3, T4), then the D3 = JSD(T5, T6). And get the average of (D1, D2, D3)?? (JSD stands for the Jensen-Shannon divergence) – just_learning Feb 18 '22 at 10:39
  • 1
    @just_learning No prob (I am the only person answering so far lol-others are welcome ofc). If you have multiple distributions and if you are doing a hypothesis test, I would ask what exactly is your null hypothesis? e.g. is it that all samples are drawn from the same distribution? – Golden_Ratio Feb 18 '22 at 10:42
  • @Golden_Ratiο: Ok! :-) Yes from the same distribution... (But if it is possible I would like to now what your suggestion for different distribution...) – just_learning Feb 18 '22 at 10:49
  • 1
    One possibility is do multiple pairwise Kolmogorov Smirnov tests for each pair of histograms (for 6 samples, you would do ${6\choose 2}=15$ tests). The only caution with multiple testing is you cannot simply compare each p value with your type 1 error $\alpha$ due to the multiple testing problem. There are procedures to control the familywise error rate, such as the Bonferroni method and, even better, the Holm method(see here) – Golden_Ratio Feb 18 '22 at 11:00
  • 1
    But I'm not sure now what you want to do, since you mentioned JS divergence in your previous reply. JS divergence is measuring how different two histograms are, but again, I wouldn't suggest using it for testing whether they are similar (I am not aware of a test that uses JS divergence).. – Golden_Ratio Feb 18 '22 at 11:04
  • 1
    Also, here are some further lecture notes on Kolmogorov Smirnov if it helps: https://www.astro.umd.edu/~miller/teaching/astrostat/lecture09.pdf – Golden_Ratio Feb 18 '22 at 11:07
  • I have transformed JSD to similarity like this: 100 - JSD*100. I have the T1 histogram as base and I compare with T2, then I compare T1 with T3, then I compareT1 with T4, etc. What I want to find out is that if T1 histogram is similar (not the same) to the other histograms or it is completely different... – just_learning Feb 18 '22 at 11:10
  • 1
    @just_learning That's fine to report the percentage similarity as such for each pair of histograms. But if you want to make statements such as "T1 and T2 are similar," "T3 and T5 are not similar", etc. I wouldn't just use JS divergence, because how would you decide the cutoff %similarity before you claim they are "similar"? The point of using a formal test is that you don't have to decide the cutoff, as the test already prescribes the cutoff to use based on the type 1 error you want to achieve. – Golden_Ratio Feb 18 '22 at 11:16
  • Ok! So, that's why you suggest (above) Kolmogorov Smirnov, Holm and multiple testing problem ? – just_learning Feb 18 '22 at 11:22
  • 1
    @just_learning exactly. hope I didn't confuse you too much lol – Golden_Ratio Feb 18 '22 at 11:24
  • I am using KS on 2 histograms and I take this: Kolmogorov-Smirnoff...KstestResult(statistic=0.3333333333333333, pvalue=0.9307359307359307). How do I transform it to percentage of similarity? – just_learning Feb 19 '22 at 12:25
  • 1
    @just_learning Well the KS statistic itself gives you one metric; it is always between 0 and 1 and is larger when distributions are less similar, so 1-KS statistic gives you a metric for similarity. It is a different measurement than 1-sqrt(JS divergence). – Golden_Ratio Feb 19 '22 at 13:07
  • 1
    @just_learning see https://stats.stackexchange.com/questions/82076/similarity-measure-between-multiple-distributions and https://medium.com/geekculture/techniques-to-measure-probability-distribution-similarity-9145678d68a6 – Golden_Ratio Feb 19 '22 at 13:07
  • I have been studying all these, the current days. But I cannot understand how to pick the threshold, above which the similarity is accepted and below which the similarity is not accepted. I believe that this has to be picked empirically. Am I losing something here? – just_learning Feb 21 '22 at 13:52
  • 1
    @just_learning How would you pick it "empirically"? – Golden_Ratio Feb 22 '22 at 00:35
  • @Golden_Ratiο: Empirically does not work, I mean to put a threshold by running many tests and see where to put it. What's your opinion about my problem? How to pick a threshold? Also what does this mean "The point of using a formal test is that you don't have to decide the cutoff, as the test already prescribes the cutoff to use based on the type 1 error you want to achieve."? As I understand, the threshold is picked "automatically" via the implementation of the related algorithm? I am new to field! So sorry for asking too many! – just_learning Feb 24 '22 at 09:45
  • 1
    @just_learning MSE is not really the best place to have these extended discussions, especially since I don't know your background knowledge. Typically, I would discuss these things over tutoring sessions/classes I hold. If you have an academic advisor, perhaps you can try reaching out to them? I am not sure I have much to add besides the above remarks/links. – Golden_Ratio Feb 24 '22 at 13:10