2

I have written a script to analyze some data at work, and for each run, it outputs a long list of integers. Each set of results is a frequency distribution (each integer appears one to many times). The result set also includes two particular integers that I have to automatically identify (called a co-pair). [P.S. The "co-pair" is just a term my supervisor uses, it is not a standard mathematical term. ]

I tested the script with smaller data sets where I already know what the co-pair should be, and realized that neither of the integers in the co-pair are always the most frequent or the least frequent in the set. Now I am at a loss what other way to statistically examine the result set and automatically find the co-pair for ANY set of data.

I have a relatively weak background in statistics so I am hoping someone can point me in the right direction.

Edit: More context.

Essentially after analyzing my data I have many sets of frequency distributions and I want to compare all of the sets at once to find answers "defining the co-pair", such as: (a) "the co-pair are ALWAYS at ogive 35 and ogive 80" or (b) "the co-pair are ALWAYS the mode and the least frequent number in each set" etc. The solution is definitely not either of the above, but what statistical methods can I use to compare many data sets to explore the relevance of particular needles in each haystack, to get an answer that works for identifying the co-pair within ALL the sets, an answer like either (a) or (b)?

Edit two: I have thousands of data sets where I already know what the co-pairs are. I want to statistically analyze them and research what possibly defines them within each set, so I can apply the same methods to the other millions of data sets where I DO NOT know the co-pairs.

Cogicero
  • 167
  • I do not see an explanation for what a co-pair is that a human could understand. A Google search doesn't immediately find anything relevant. I cannot tell if your first paragraph describes a set of training data for your script that learns from the example (dataset, dataset, distribution, co-pair) collections so as to be able to find co-pairs in the future. ... or is there some definition of co-pair that your script should implement that does not appear in your question? – Eric Towers Jun 27 '14 at 08:01
  • Thanks Eric. The "co-pair" is just a term my supervisor uses, it is not a standard mathematical term. No, the script does not learn from any example. I am asking what statistical methods that I can research to identify a number which has a particular (unknown) relevance within each result set. I have tried the basic ones: mean, median and mode. – Cogicero Jun 27 '14 at 08:07
  • @Cogicero - if you can't explain what a co-pair is, how do except us to find it? – Nathaniel Bubis Jun 29 '14 at 22:20
  • @nbubis: Don't mind the name "co-pair", it is just a name. My question is something like this: after analyzing data I end up with 1,4,13,13,2,7,13,5,3,8,8,8,8,2,3,19,10,19,2,8,5,2,3,4,1,10 and the numbers I am meant to fish out are 13 and 19, but I do not know them yet. The "co-pair". Another set of data will give ANOTHER set of numbers after analysis, with its own co-pair. I have a LOT of data to analyze, and find a method to identify that "unique" pair in all of them. One such statistical function is a MODE. (But in this case it is obviously not the mode I'm after). I hope it's clearer. – Cogicero Jun 29 '14 at 22:25
  • @Cogicero - no you're not clear. You've given no definition of how those numbers should be chosen, so there is no way to find them. – Nathaniel Bubis Jun 29 '14 at 22:50
  • @nbubis: I have thousands of data sets where I already know what the co-pairs are. I want to analyze them and find what defines them in each set, so I can apply the same methods to the other millions of data sets where I DO NOT know the co-pairs. P.S. I also updated the question. Thanks. – Cogicero Jun 29 '14 at 22:52

1 Answers1

1

The methods used to solve such problems are generally termed Machine Learning. But, for you to use these methods, you need to first postulate a model of how you think these "co-pairs" are chosen, based on your understanding of what this data actually represents.

One general method you could use is if the the data you get for each run has constant length (if not, you can always make it so using re-sampling methods). Now you have a set of $N$ vectors of length $M$ (where $N \gg M$), you want to find for each a pair of numbers $x,y$.

You can then attempt to run a classifier method such as SVM to find, given a new vector of length $M$, what the the probabilities are to get numbers $x,y$.

Machine learning is a complex and delicate topic, that does not lend itself very well to "plug & play" programming. You'll probably need to spend a lot of time studying it before getting good results, and if this problem is fundamental to your company, you may want to use the services of an expert in the field.

Nathaniel Bubis
  • 33,018
  • Hi, I have accepted your answer. Thanks. I am not searching for a Machine Learning method though. I imagine there should be a way to plot graphs of each set of results, then superimpose them on one another to discover the pattern. I am just wondering what kind of plots I will employ. I'll keep trying. – Cogicero Jul 02 '14 at 04:34
  • 1
    @Cogicero - Machine Learning is what is generally used to find an unknown pattern, unless it's something very simple, which in that case multiple plots may give you a clue. Keep in mind that real data is often not as simple as you may think :) – Nathaniel Bubis Jul 02 '14 at 04:37