If I am 80% sure, is there an 80% chance that I'm right?

Question

A friend of mine said this, and I couldn't argue why I thought it was wrong.

Obviously, in some circumstances it is valid, say if you flip a coin, and I say I'm 50% sure it lands on heads.

In a discussion about whether sending a "dick pic" was illegal, he said:

I'm 99.3% sure, so there's only a 0.7% chance that I'm wrong.

Now, if we assume that it is illegal, I would agree, but isn't there a 50% chance that there is a 99.3% chance that he is wrong?

Could someone explain to me if and how this is wrong?

Answer 1

I think the main problem here is that "he is sure" is something else than "he is right". He might be 100% sure but still dead wrong, while when there is a 100% chance that he is right, then he is right.

In the sentence you highlighted, he is assuming that his being sure is equivalent to his being right. Of course, a sentence like

The chance that I am right is 99.3%, so there is only a 0.7% chance that I'm wrong.

is completely correct, but he doesn't know the chance that he is right, he just gives some random, high number to convince you.

Answer 2

A probability is a measurement about your current beliefs and confidence, not an objective statement about the outside world.

Suppose a dealer picks a playing card out of a deck in front of you and me, and hides the card from view. The dealer asks for the probability that this card is the ace of diamonds.

1. Because I didn't see the card at all, it's right for me to say that I believe there's a 1/52 chance that the card is the ace of diamonds.

2. You accidentally saw the card as it was being drawn, but only enough to see that it was a red card. It's right for you to say that you believe there's a 1/26 chance that the card is the ace of diamonds.

3. The dealer has read the card. It's right for the dealer to say that there's a 100% chance (or a 0% chance) of the card being the ace of diamonds.
4. A fourth person joins the table and for no apparent reason is suddenly struck by the idea that the card really really ought to be the ace of diamonds. The newcomer announces that the probability of the card being the ace of diamonds is 99.3%.

There is a different correct probability assignment for each person. Each person expects to be right a certain fraction of the time, but this is on the basis of different amounts of knowledge. You might even be like the newcomer, and assign a degree of confidence which is unwarranted by the data you have.

I would expect that if a person assigns the right probability corresponding to their level of knowledge (like the first three people), that person should be correct that same fraction of the time. If, however, a person assigns probabilities that don't respect the information they have (like the fourth person), there's no guarantee that expectations will agree with reality.

Answer 3

The way I read it your friend claims he has the following 'superpower':

Of all the gazillion times (over the course of his entire life) that he claims he is 99.3% sure of something, in 99.3% of cases it later turns out he is actually right. Similarly, in all of the bazillion cases (over the course of his life) he claimed to be 60% sure of something he turned out to be right in 60% of the cases and similar for 10%, 100%, 5% etc.

Now two things can be said about this super-power.

1. If one insists on making statements about levels of certainty this is an incredibly useful superpower to have or at least to strive for. It provides the statements with a very natural interpretation that makes it easy for the listener to judge its value.

2. Humans are generally very bad at estimating probabilities and so the chances that your friend actually has the above superpower are rather slim.

However your friend is not the only one making statements of this kind. The National Cancer Institute in the United States for instance has a 'tool' that, based on a number of questions about a person produces a probability that she will develop breast cancer. (Link: https://www.cancer.gov/bcrisktool/). This is an example where the makers strive to comply with what I call the 'superpower' above. I.e. the claim (on the 'about' page of the website) that the model "has been shown to provide accurate estimates of breast cancer risk" means that among women for which it predicts X% risk, a number close to X% actually develops breasts cancer within the given time interval, and that for all X.

Of course one needs criteria how close 'close to X%' should be to call the model 'accurate' and how much this closeness or lack thereof may vary depending on X. There are various answers to this question, some of which are discussed in the references in the 'about' section of the webpage.

Answer 4

If a statement is true with $80$% certainty, then out of every $100$ trials, the statement will be true around $80$ times. So yes, there's an $80$% chance that the statement is true in a given trial.

Answer 5

This question boils down to the statistical concept of calibration. A system that makes predictions (i.e. your friend) is well-calibrated if its probabilistic predictions are borne out by empirical evidence. Without knowing how well-calibrated your friend is, it is impossible to say if his estimates of probability are reliable. Certainly him stating a probability estimate doesn't make it so, we need more information about if those estimates are any good or not.

For example, if your friend makes the statement that he is 75% sure of something, and is actually correct 75% of the time of these predictions, he is well-calibrated in this range. On the other hand, if events that he predicts will happen with 30% probability actually occur 50% of the time, he tends to underestimate likelihood and is not well-calibrated.

If your friend is well-calibrated, then yes, his statements of likelihood are in fact very close to the true likelihood of an event occurring. If he is 99.7% sure of something, then he will only be wrong 3 times in 1000. However, if your friend is poorly calibrated, his estimates of likelihood don't really have any bearing on reality. He could be 99.7% sure of something and still have it occur 0% of the time.

You can test your friend's calibration with a simple online test, found at http://calibratedprobabilityassessment.org/. This tests asks a series of general knowledge questions. The goal isn't to answer them all correctly, but to correctly judge your confidence in your answers. For each question, you must select one of two options and rate your condfidence in the answer from 50-100%. A well-calibrated person will get roughly 60% of the questions right that they are 60% sure of, 80% of the questions right that they are 80% sure of, and 100% of the questions right that they are certain of.

Answer 6

Saying that you are "sure" something is true, is not the same as it actually being true.

In cases where something is true or false, and someone makes a guess of the outcome there is a 50% chance they will be correct, and a 50% chance they will be wrong/

In this case if your friend says he is "99.3% sure" that the answer to the statement is "true". However there is nothing to say it is actually true - it is merely a hunch that he has.

The chance of him being right is still 50%.