AP® Statistics · Unit 3 · Topic 3.6
Significance Level and the Decision
Significance level alpha is the cutoff you set in advance for how small a p-value must be before you call the result convincing; comparing p-value to alpha turns evidence into a decision.
Alpha is a line drawn before you see the data. Common choices are 0.05 or 0.01, and it represents how much risk you're willing to accept of rejecting a true null hypothesis just because of random sample-to-sample variation. There's nothing magic about 0.05: it's a convention, a shared agreement about how rare a result has to be before it stops looking like luck and starts looking like evidence against the null.
The decision rule is purely mechanical once alpha is chosen: if the p-value is less than alpha, reject H0, because the data would be surprisingly rare if the null were true. If the p-value is greater than or equal to alpha, fail to reject H0, because the data isn't rare enough to rule out the null as a plausible explanation. The p-value measures how surprising your sample result is; alpha decides how much surprise is enough to act on.
This decision connects directly to the two kinds of errors a test can make. Reject H0 and you risk a Type I error, rejecting a true null. Fail to reject and you risk a Type II error, missing a false null. Neither decision proves anything: failing to reject just means the evidence wasn't strong enough, and rejecting just means the null looks implausible, not that the alternative is confirmed true. The decision is always a statement about evidence strength relative to a chosen standard, never a certainty.
Key terms
- Significance level (alpha)
- The pre-chosen probability threshold, often 0.05, that a p-value must fall below to reject the null hypothesis.
- p-value
- The probability of getting a sample result at least as extreme as the one observed, assuming the null hypothesis is true.
- Reject H0
- The decision made when the p-value is less than alpha, indicating the data is unusually inconsistent with the null.
- Fail to reject H0
- The decision made when the p-value is greater than or equal to alpha, indicating insufficient evidence against the null.
- Type I and Type II error
- Type I is rejecting a true null; Type II is failing to reject a false null.
Watch it explained
- How it worksHow does comparing a p-value with alpha turn a number into an actual decision?
- Worked exampleShow me the decision for a p-value of 0.032 at alpha 0.05 and then at alpha 0.01.
- Common mistakeWhy isn't failing to reject the null the same as proving the null is true?
- Common mistakeWhy isn't rejecting the null hypothesis the same as proving the alternative is true?
Check yourself
End of topic.