AP® Statistics · Unit 1 · Topic 1.13
What an Experiment Lets You Conclude
Random assignment lets you say a treatment caused an effect; random selection lets you say that effect applies beyond the study. An experiment can give you one, both, or neither, and your conclusion can only claim what you actually earned.
Two separate design choices buy two separate rights. Random assignment means every subject had an equal chance of landing in each treatment group, so the groups start out alike on everything, known or unknown, except the treatment itself. If the groups end up different, the treatment is the only systematic explanation left, so you get to say 'caused.' Random selection is a different move entirely: it means the subjects were drawn randomly from some larger population, so the sample looks like that population. That buys you the right to generalize results beyond the people you actually studied. One controls internal validity, the question of cause inside the study. The other controls external validity, the question of reach outside it.
Most real experiments only get one of these. A study that recruits volunteers and then randomly assigns them to treatment and control can still conclude causation, because the random assignment is doing its job. It just can't claim the result holds for everyone, because the volunteers probably aren't representative of any larger population. Flip it around: a random sample from a population, observed without random assignment, can describe that population accurately but can't establish cause.
Causation and significance are also separate ideas worth not tangling together. A result can be statistically significant, meaning unlikely to be due to chance, while the actual effect size is tiny and useless in practice. Significance answers 'is this real,' not 'does this matter.' This connects back to the sampling and assignment methods from earlier in the unit: every conclusion you're allowed to draw traces directly back to how the data was produced.
Key terms
- Random assignment
- Randomly placing subjects into treatment groups so the groups are comparable, which supports a causal conclusion.
- Random selection
- Randomly drawing subjects from a larger population, which supports generalizing results to that population.
- Statistical significance
- A result unlikely to have occurred by chance alone, which does not by itself mean the effect is large or meaningful.
- Internal validity
- The strength of a causal conclusion within the study itself.
- External validity
- How well a study's results generalize beyond the subjects actually studied.
Watch it explained
- How it worksHow do random assignment and random selection each control a completely different problem?
- Common mistakeWhy isn't a statistically significant experiment automatically important in real life?
- Going furtherWhat changes about your conclusion when the subjects were volunteers, not a random sample?
- Ties togetherHow does the scope of a conclusion depend on how subjects were chosen and assigned?
Check yourself
End of topic.