Common Statistical Fallacies
Learn to spot and avoid major statistical fallacies: regression to the mean, Simpson's paradox, law of small numbers, cherry picking, and more.
Statistical Thinking Gone Wrong
Statistics is powerful, but it's also easy to misuse — sometimes accidentally, sometimes deliberately. Statistical fallacies are errors in reasoning about data that lead to false conclusions.
Understanding these fallacies makes you:
- A better analyst (avoiding your own mistakes)
- A critical consumer (spotting manipulation)
- A clearer communicator (honest reporting)
Let's explore the most common and dangerous statistical mistakes.
Regression to the Mean
Extreme observations tend to be followed by more moderate ones just by chance, not because of any intervention.
If you're at an extreme, random variation is more likely to move you toward the average next time.
Athletes who appear on the cover of Sports Illustrated often have worse performance afterward. Is the magazine cursed?
No. Athletes make the cover after exceptional performance (far above their average). By chance alone, their next performance will likely be closer to their true average — regression to the mean.
Another example: Children of very tall parents tend to be tall, but shorter than their parents on average. Not because of bad genes — it's regression to the mean. Extreme heights partially due to luck in gene combinations average out over generations.
Why it matters: People attribute regression to the mean to interventions or causes:
- "I got sick, tried this remedy, and got better!" (You were probably at your worst — improvement expected)
- "I praised poor performers and they got worse; I criticized good performers and they improved!" (No — both regressed to the mean)
Solution: Use control groups! Only by comparing to those without the intervention can you separate real effects from regression to the mean.
Simpson's Paradox
A trend appears in different groups but reverses when groups are combined (or vice versa).
An association can disappear or reverse when you account for a confounding variable.
UC Berkeley was sued for gender bias in 1973. The data:
- Men: 44% admitted
- Women: 35% admitted
Looks like discrimination! But when broken down by department:
Department A:
- Men: 62% admitted (825/1340)
- Women: 82% admitted (108/131)
Department B:
- Men: 63% admitted
- Women: 68% admitted
In EVERY department, women had equal or higher admission rates!
What happened? Women applied to more competitive departments (lower admission rates overall). Men applied to less competitive departments.
Aggregate numbers misleading. The apparent bias disappeared when controlling for department.
Simpson's Paradox teaches: Always disaggregate data and look for confounders. Aggregate statistics can hide or invent trends that don't exist at finer levels.
Law of Small Numbers
People intuitively expect small samples to behave like large samples. They don't.
Small samples have high variability. Patterns in small samples are often just noise.
A small town has 10 cancer cases out of 500 people (2%) vs national rate of 1%.
"There must be an environmental cause!"
But: With only 500 people, we expect 5 cases on average. Observing 10 isn't that rare — random variation can easily produce this.
Counties with the HIGHEST cancer rates are often small rural counties. Counties with the LOWEST rates are also small rural counties.
Why? Small populations have high variability. Some get unlucky, some get lucky. We notice the unlucky ones and search for causes that don't exist.
Kahneman & Tversky famously demonstrated: People believe small samples are as representative as large samples. They're not! Always check sample size before drawing conclusions.
Cherry Picking (Selection Bias)
Cherry picking: Selecting data that supports your conclusion while ignoring data that doesn't.
This can be deliberate (fraud) or accidental (confirmation bias).
"My strategy picked stocks that outperformed the market by 30%!"
What they don't tell you: They tested 100 strategies and reported the one that worked best. The other 99 failed.
This is data dredging or p-hacking — running many analyses and reporting only the significant ones.
Survivorship bias: Mutual fund companies advertise only successful funds. The failures quietly disappear from records.
How to spot it:
- Are they showing all the data or just some of it?
- Did they pre-register hypotheses or test many things?
- Are negative results hidden?
Solution: Pre-registration, full reporting, replication.
Prosecutor's Fallacy
Confusing P(Evidence|Innocent) with P(Innocent|Evidence).
Bayes' Theorem violation — ignoring base rates.
"The defendant's DNA matches. The probability of a random match is 1 in 1,000,000. Therefore, the probability they're innocent is 1 in 1,000,000."
Wrong!
P(match | innocent) = 1/1,000,000 (true)
But P(innocent | match) depends on the BASE RATE of guilt!
If you test 10 million people, you expect 10 false matches. If only 1 person is guilty, most "matches" are false positives!
Prosecutor's fallacy has led to wrongful convictions. You must account for prior probability (base rate).
Ecological Fallacy
Inferring individual-level relationships from group-level data.
"Rich states vote Democrat, so rich people vote Democrat."
Wrong! At the state level, wealthier states lean Democrat (California, New York). But WITHIN states, wealthier individuals lean Republican.
The fallacy: Group-level patterns don't necessarily hold at individual level. California is rich and Democrat because of its wealthy liberals, not because wealth causes Democrat voting.
Texas Sharpshooter Fallacy
Named after a joke: A man shoots randomly at a barn, then paints targets around the bullet holes and claims he's a sharpshooter.
Defining patterns AFTER seeing the data, then acting like you predicted them.
"I predicted the stock market crash! Look, my indicators showed warning signs."
But: You defined those "indicators" after the crash. With enough data, you can always find something that "predicted" it in hindsight.
Hindsight bias makes past events seem predictable when they weren't.
McNamara Fallacy
Making decisions based solely on quantifiable metrics while ignoring everything that can't be measured.
"If you can't measure it, it doesn't exist."
U.S. Secretary of Defense Robert McNamara used enemy body counts as the primary metric of success in Vietnam.
Problem: This metric was gameable, misleading, and ignored strategic reality. High body counts didn't equal winning.
Modern examples:
- Teaching to standardized tests (ignoring creativity, critical thinking)
- Optimizing only for metrics you can measure (clicks, views) while ignoring quality
- Judging employees only on quantifiable KPIs
Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure."
Not everything that can be counted counts, and not everything that counts can be counted.— Albert Einstein
Defending Against Fallacies
1. Always ask for sample size Small samples → high noise. Don't trust patterns from n < 30.
2. Look for control groups "X happened after Y" doesn't mean Y caused X. What happened to those without Y?
3. Check for cherry picking Are they showing all the data? What about failed attempts?
4. Remember base rates Rare events stay rare even with impressive-sounding evidence.
5. Disaggregate when possible Aggregate statistics hide confounders (Simpson's Paradox).
6. Question metrics What's NOT being measured? Can the metric be gamed?
7. Demand replication One study proves nothing. Replication and meta-analysis matter.
8. Think about incentives Who benefits from this conclusion? Follow the money.