2

I have 10 databases of facts from 10 different sources, I now know if the facts were true or false.

What is the best way to form an equation to determine the reliability of each of the source?

I don't want a source who got 4/4 True to have a higher reliability rating than someone who got 395/400 correct. I want a formula that takes into account the total number of facts in the database ?

Thanks for any help.

Irregardless
  • 469
gweno10
  • 107

1 Answers1

1

One way to do this (the way I would choose) would be a Bayesian approach that starts with a prior belief that each source is (e.g.) 50% reliable, and then let the data update your prior belief.

Specifically, we let the prior reliability of each source be the mean of a $\text{beta}(1,1)$ random variable -- equivalently, a standard uniform random variable. Then, for each source, you let its posterior reliability be distributed as $\text{beta}(1+A,1+N-A)$, where $A$ is the number of true responses from that source, and $N$ is the total number of responses (so $N-A$ is the number of false responses).

You now have a probability distribution on the reliability of each source taking into account your data. As you requested, it is sensitive to volume of responses, so that 395/400 is much weightier than 4/4. If you want a point estimate of the reliability of each source, then you can take the posterior mean of the distribution of each source. The mean of a $\text{beta}(\alpha,\beta)$ random variable is given by $\frac\alpha{\alpha+\beta}$.

So, the point estimate of the posterior reliability of the 4/4 source would be $\frac56$, and for the 395/400 source it would be $\frac{396}{402}$.

So, in summary, to get the reliability of a source with $A$ true responses out of $N$ total responses, you take $\frac{A+1}{N+2}$.

This approach can be altered to start with different prior beliefs than $50\%$ reliability. You could even assign different prior reliabilities to different sources, if some of them had independent reason to be treated as safe or as unsafe. Furthermore, you could also tune the model to be "less impressed" by the data, so that it takes more data to move you away from the prior reliability. To do so, you would start with a $\text{beta}(a,a)$ for some $a$ higher than 1. Whether you'd want to do something like that depends on the particularities of your problem.

Edited to add: Another benefit of this method is that it gives you uncertainty estimates for your posterior reliabilities. That is: not only is $395/400$ a more impressive score than $4/4$, the volume of data in the former case also makes you much more confident of your estimate than in the $4/4$ case. The variance of a $\text{beta}(\alpha,\beta)$ random variable is given by $\frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)}$. So, in the case where you start with a prior of $\text{beta}(1,1)$ and a source gets $A$ true responses out of $N$ total, the variance of its posterior reliability $r$ is given by $$\text{Var}(r) = \frac{(A+1)(N-A+1)}{(N+2)^2(N+3)}$$

Ceph
  • 1,336
  • Somehow I am not quite happy with describing a prior or posterior distribution as a point estimate. In a sense you acknowledge this towards the end, where your $\text{beta}(a,a)$ prior corresponds to "$50%$ reliability" for all values of $a$ even though different $a$s will lead to different effects which combined with the observations – Henry Jul 18 '17 at 18:56
  • I'm not certain about the nature of your unhappiness -- maybe you can explain more? But I think I agree that it would be more correct to say "prior distribution on the reliability of each source with prior mean at 50%" where I instead simply said "prior belief that each source is 50% reliable", if that is what you are saying. I am uncertain about OP's level of knowledge of Bayesian stats and I was sacrificing a bit of rigor for ease of comprehension. – Ceph Jul 18 '17 at 19:02