Trading Off the Two Errors

A hypothesis test uses a cutoff to decide when evidence is strong enough to reject the null hypothesis. That cutoff sits on a curve of possible sample results. Move it to demand stronger evidence before rejecting, and you shrink the rejection region, the area that gives Type I errors, rejecting a true null. But everything you no longer reject gets lumped into the other region, so the area for Type II errors, failing to reject a false null, grows. With sample size fixed, the two error rates sit on a seesaw: push one down and the other rises.

The only way to shrink both at once is to collect more data, which sharpens the distributions so there's less overlap to cause either kind of mistake. Short of that, the cutoff is a tradeoff, and where you set it should depend on which error costs more. A smoke alarm maker who sets alpha very low gets very few false alarms, but pays for it with more missed real fires, a Type II error, since the sliding cutoff that cuts false alarms also makes the alarm harder to trigger for genuine smoke. A disease screening test, by contrast, usually accepts more false positives to avoid the much costlier false negative.

This tradeoff is why alpha is chosen deliberately rather than just set to the smallest possible number, and it sets up power, the probability of correctly rejecting a false null, which is just 1 minus the Type II error rate and rises whenever Type II error falls.