What a p-Value Actually Measures

A p-value answers a question about the data, not a question about the hypothesis. The question it answers is: if the null hypothesis were true, how often would a sample like mine, or one even further from what the null predicts, show up just by chance? That's a statement about how surprising the data is, assuming the null. It never flips around to tell you the probability that the null itself is true.

Picture the null distribution, the bell-shaped curve of all the sample proportions or statistics you'd expect if the null hypothesis were exactly right. Your actual test statistic lands somewhere on that curve. The p-value is the area under the curve out past that point, in the tail, including everything at least as extreme. Shade that tail, find its area, and you have the p-value. Push the test statistic further from the center, out into the tail, and that shaded area shrinks, because extreme results get rarer under the null the further out you go.

This is why a p-value of 0.03 is not a 3 percent chance the null hypothesis is true, and not a 3 percent chance someone made a mistake. It is a 3 percent chance of seeing data this extreme or more, purely from random sampling variation, if the null were correct. Small p-values make the null look like a bad explanation for the data, which is the bridge into deciding whether to reject it and into the two kinds of errors that decision can produce.