Power of a Test

A test can miss a real effect simply because it didn't look hard enough. If the true proportion is different from the null value but only by a little, or if the sample is small, the sample results can easily look consistent with the null just by chance. Power is the probability that the test correctly rejects the null hypothesis when the null is actually false. Low power means a real effect is likely to slip through undetected, a Type II error.

Picture two sampling distributions of the sample proportion: one centered at the null value, one centered at the true value, which is different. The rejection region is a cutoff drawn from the null distribution. Power is the area of the true distribution that falls inside that rejection region, meaning power shows up as overlap between two shifted curves. The less these two distributions overlap, and the more of the true distribution sits inside the rejection region, the higher the power.

Power connects directly to the tradeoffs already built into a test. Increasing sample size shrinks the standard error, which tightens both distributions and pulls them apart relative to their width, raising power. Increasing alpha widens the rejection region, catching more of the true distribution but also raising the Type I error rate. A bigger true effect pulls the two distributions further apart, which also raises power. Power, Type I error, and Type II error are three sides of the same tradeoff: you can't minimize all three at once without changing the sample size.