Independence
Master the concept of statistical independence, the crucial difference between independence and mutual exclusivity, conditional independence, and why it matters for modeling.
What Does Independence Really Mean?
Two events are independent if knowing one happened tells you nothing about whether the other happened. The outcome of one doesn't change the probability of the other.
This is one of the most important concepts in statistics because:
- Most statistical models assume independence
- Violations of independence are a major source of errors
- Understanding dependence helps us model real-world relationships
Independence is both simpler and more subtle than it first appears.
The Formal Definition
Events A and B are independent if and only if:
Equivalently (and often more intuitively):
Translation: Knowing B happened doesn't change the probability of A, and vice versa.
Coin flip and die roll:
- P(Heads) = 1/2
- P(rolling 6) = 1/6
- P(Heads AND 6) = 1/2 × 1/6 = 1/12 ✓
These are independent — the coin has no influence on the die.
Weather in New York and a coin flip: Independent. Your coin doesn't control the weather.
Drawing cards without replacement:
- P(1st card is Ace) = 4/52
- P(2nd card is Ace | 1st was Ace) = 3/51 ≠ 4/52
These are dependent — the first draw changes the deck for the second.
Rain and wet streets: Highly dependent. If it's raining, streets are almost certainly wet.
Independence vs Mutually Exclusive
This is a common confusion that trips up students constantly. These are NOT the same thing!
Property | Independent | Mutually Exclusive |
|---|---|---|
| Definition | P(A∩B) = P(A)·P(B) | P(A∩B) = 0 |
| Meaning | Knowing A doesn't affect B | A and B can't both happen |
| Example | Coin flip and die roll | Drawing a King and a Queen (same card) |
| Can both occur? | Yes | No — that's the point |
| Relationship | Knowing one tells you nothing | Knowing one tells you the other didn't happen |
Key insight: If A and B are mutually exclusive and both have positive probability, they cannot be independent!
Why? If P(A) > 0 and P(B) > 0 but they're mutually exclusive:
- P(A∩B) = 0
- P(A)·P(B) > 0
- So P(A∩B) ≠ P(A)·P(B)
Knowing A happened tells you B definitely didn't — that's dependence!
Common mistake: "These events are mutually exclusive, so they're independent." This is exactly backward. Mutually exclusive events with positive probability are maximally dependent.
Testing for Independence
How do you check if two events are independent in practice?
Does a new drug prevent heart attacks? We observe:
- 1,000 people took the drug → 20 had heart attacks (2%)
- 1,000 people took placebo → 40 had heart attacks (4%)
Are "taking the drug" and "heart attack" independent?
If independent: P(heart attack | drug) should equal P(heart attack | placebo)
But 2% ≠ 4%, so they're dependent (which is good — the drug works!).
Independence would mean the drug has no effect. Finding dependence is the whole point of the study.
In general, to test independence of events A and B from data:
- Calculate P(A|B) and P(A|not B)
- If they're different (beyond random noise), A and B are dependent
- Statistical tests (like chi-square) formalize this comparison
Conditional Independence
Sometimes events are dependent unconditionally but become independent when you condition on a third event.
A and B are conditionally independent given C if:
P(A ∩ B | C) = P(A|C) · P(B|C)
Consider:
- A = "Person has large vocabulary"
- B = "Person has large shoe size"
These are positively correlated! Why? Because:
- C = "Person is an adult" vs "Person is a child"
Adults have both larger vocabularies and larger shoe sizes than children.
But given age (conditioning on C):
- P(large vocab | adult AND large shoes) ≈ P(large vocab | adult)
Shoe size tells you nothing extra about vocabulary once you know the age. They're conditionally independent given age, even though they're dependent overall.
Age is a confounding variable that creates spurious correlation between vocabulary and shoe size.
This is the foundation of causal reasoning. Many apparent relationships disappear when you control for confounders. This is why randomized controlled trials are so powerful — they eliminate confounding.
Why Independence Matters in Modeling
Most statistical methods assume observations are independent. This assumption is called i.i.d. (independent and identically distributed).
When independence fails:
1. Time Series Today's stock price depends on yesterday's. Past observations predict future ones. Standard methods fail; need specialized time series models.
2. Clustered Data Students within the same classroom are more similar than random students (they share a teacher, curriculum, etc.). Treating them as independent underestimates uncertainty.
3. Network Data Your friends' opinions influence yours. Social network data violates independence assumptions.
4. Repeated Measures Measuring the same person multiple times creates dependence. Person A's measurements are more similar to each other than to Person B's.
Bad survey design: Survey 100 people at a Trump rally about their voting preference. Get 95% Trump support.
The problem? These observations aren't independent — they're all people who chose to attend a Trump rally. They're more similar to each other than to random voters.
This is called sampling bias, and it happens because independence is violated.
Critical for valid inference: If your data violates independence and you pretend it doesn't, your p-values will be too small, confidence intervals too narrow, and conclusions overconfident. Always check if independence is reasonable.
Pairwise vs Mutual Independence
Pairwise independence: Every pair of events is independent. Mutual independence: All subsets are independent.
For three events A, B, C to be mutually independent:
- P(A∩B) = P(A)·P(B)
- P(A∩C) = P(A)·P(C)
- P(B∩C) = P(B)·P(C)
- P(A∩B∩C) = P(A)·P(B)·P(C) ← Also need this!
Pairwise independence doesn't guarantee mutual independence.
Flip two fair coins. Define:
- A = "First coin is Heads"
- B = "Second coin is Heads"
- C = "Exactly one coin is Heads"
Check pairwise independence:
- P(A) = P(B) = P(C) = 1/2
- P(A∩B) = 1/4 = P(A)·P(B) ✓
- P(A∩C) = 1/4 = P(A)·P(C) ✓
- P(B∩C) = 1/4 = P(B)·P(C) ✓
But: P(A∩B∩C) = 0 (can't have both heads AND exactly one head) While P(A)·P(B)·P(C) = 1/8 ≠ 0
Pairwise independent but not mutually independent!