I understand the alpha is 1-the confidence level. I heard 2 things about the alpha aka the significance level, and they seem to contradict in my view. They are:
Alpha is the probability of incorrectly rejecting the null hypothesis ie saying something is happening when there isn't.
Alpha is the standard for how extreme you information must be before you can reject the null hypothesis ie how frequently the event you observed happens by chance alone.
Say you think you think Drug A works against a disease and that alpha is 5%. The null hypothesis says, nothing unusual is happening ie, the drug has no special effect on the disease.
By definition 1, alpha would mean the chances of you saying the drug works when it actually has no effect, is 5 times out of 100.
By definition 2, you are saying, the event you observed (that the drug cured a disease) must happen by chance only 5 times out of 100 times.
Definition 1 is talking about you making a mistake (saying there is something when there isn't, ie a false positive). Definition 2 is talking about the alpha level itself. I find it contradictory because if it is talking about you or your diagnostic test, it has such a low chance of making a mistake of saying there is something special/strange happening when there isn't (5% chance of making a type 1 error), that is good, since it means you are right most of the time when you say there is nothing special happening ie, most of the time, the drug doesn't work, and you made the rare mistake of saying it DOES work. (or you encountered one of the rare times that it did work)
Meanwhile, definition 2 says, the observed event (that a disease was cured) happens as frequently as the significance level by chance alone, in this case, it happens 5% of the time by chance alone. If something happens only 5% of the time by chance alone, it is not very common, it is quite rare. If it is rare, it is significant: If the event (that a disease was cured) happens by chance alone 5% of the time, then that is not very frequent, it is quite rare, so, if the drug cured a disease, it probably truly DOES work, because the chances of the disease curing itself is only very rare (5% of the time). In other words, since the chance of the disease being cured randomly by chance is 5%, and adding the drug cured the disease, there IS something special happening, ie, the drug works.
I would like to know what it actually is. All other definitions or explanations confuse me more and lead me to think it is the same as the p value. Could someone explain what alpha/significance levels actually are?
