Unit 3: Statistics and the Replication Crisis
The machinery that turns noisy data into a claim you can trust, and the reckoning the field had when too many claims failed to replicate.
7 topics · 25 questions
Each topic is a path, not a menu: work down it in order and every question builds on the one above. Click any of them to watch the answer draw itself.
The Normal Distribution
- How it worksHow does a histogram of thousands of individual measurements smooth out into a normal curve?
- Worked exampleFind the percentage of scores that fall within one standard deviation of the mean on a normal distribution.
- Going furtherWhat happens to the shape of a distribution when a trait is driven by one dominant factor instead of many small independent ones?
Your turn1 of 2
About 68 percent of scores in a normal distribution fall within 1 standard deviation of the mean. If a distribution has a mean of 96 and a standard deviation of 14, find the lower bound of this middle 68 percent range.Standard Deviation as a Measure of Spread
- How it worksHow does standard deviation turn the scatter of individual scores around the mean into a single number?
- Worked exampleFind the standard deviation of the scores 2, 4, 4, 4, 5, 5, 7, 9 given a mean of 5.
- Common mistakeWhy doesn't a small standard deviation mean the average itself is more likely to be correct?
Your turn1 of 2
A small dataset has four scores: 11, 10, 14, and 5. Find the standard deviation of this dataset, dividing by n rather than n minus 1.Effect Size as Distribution Overlap
- How it worksHow does an effect size turn the overlap between two groups' distributions into one comparable number?
- Worked exampleFind Cohen's d for two groups with means 100 and 108 and a pooled standard deviation of 16.
- Common mistakeWhy isn't a statistically significant result automatically a large one?
- Going furtherWhat happens to the overlap between two distributions as the effect size climbs from 0.2 to 0.8 to 2.0?
- Ties togetherHow does an effect size viewed as distribution overlap connect to why a study needs enough statistical power to detect it?
Your turn1 of 2
Find Cohen's d for two groups with means of 98 and 111 and a pooled standard deviation of 24.Statistical Significance and the p-value
- How it worksHow does a p-value measure how surprising your data would be if there were truly no effect at all?
- Worked exampleFind why a coin flipped 20 times and landing heads 16 times gives a p-value around 0.01 against a fair-coin null hypothesis.
- Common mistakeWhy doesn't a p-value tell you the probability that the null hypothesis is true?
- Common mistakeWhy isn't p less than 0.05 a guarantee that a result is real and not a fluke?
- Going furtherWhat happens to the p-value of the exact same effect when the sample size is multiplied by ten?
Your turn1 of 2
A study's test statistic is 2 with a given sample size. If the sample size is multiplied by 10 while the underlying effect size stays exactly the same, the test statistic scales up by the square root of that factor. Find the new test statistic.Statistical Power and Sample Size
- How it worksHow does statistical power depend on sample size, effect size, and how much noise is in the measurement?
- Worked exampleFind why a study needs roughly four times as many participants to detect an effect half as large.
- Going furtherWhat happens to a field's published results when most of its studies are underpowered to detect the effects they're chasing?
Your turn1 of 2
A study needs 151 participants to reliably detect an effect size of 0.677. Because the sample size needed scales as 1 over the effect size squared, find how many participants would be needed to detect a smaller effect size of 0.333 with the same reliability.The Replication Crisis in Psychology
- How it worksHow do underpowered samples, flexible analysis choices, and publishing only positive results combine to manufacture false findings?
- Worked exampleFind what a 36 percent replication rate implies about how many of 100 published effects were probably not real to begin with.
- Common mistakeWhy doesn't a study appearing in a top journal guarantee that its effect is real?
Your turn1 of 2
A large replication project successfully reproduces 46 percent of 158 published psychology effects it attempts to replicate. Find how many of those studies failed to replicate.Publication Bias and the Funnel Plot
- How it worksHow does a funnel plot reveal publication bias by showing which side of the funnel has missing studies?
- Worked exampleFind why a field averaging effect sizes only from published studies will overestimate the true effect.
- Going furtherWhat happens to a meta-analysis's effect-size estimate once a funnel plot's missing studies are filled back in?
Your turn1 of 2
A meta-analysis of 46 published studies has an uncorrected average effect size of 0.689. A funnel-plot trim-and-fill correction adds back an estimated 24 missing null-result studies with an effect size near zero. Find the corrected average effect size, weighting all studies, published and added, equally.