1.5 KiB
1.5 KiB
Statistics Reference for Product Managers
p-value
The p-value is the probability of observing data at least as extreme as yours if there were no true effect.
- Small p-value means data is less consistent with "no effect".
- It does not tell you the probability that the variant is best.
Confidence Interval (CI)
A CI gives a plausible range for the true effect size.
- Narrow interval: more precise estimate.
- Wide interval: uncertain estimate.
- If CI includes zero (or no-effect), directional confidence is weak.
Minimum Detectable Effect (MDE)
The smallest effect worth detecting.
- Set MDE by business value threshold, not wishful optimism.
- Smaller MDE requires larger sample size.
Statistical Power
Power is the probability of detecting a true effect of at least MDE.
- Common target: 80% (0.8)
- Higher power increases sample requirements.
Type I and Type II Errors
- Type I (false positive): claim effect when none exists (controlled by alpha).
- Type II (false negative): miss a real effect (controlled by power).
Practical Significance
An effect can be statistically significant but too small to matter.
Always ask:
- Does the effect clear implementation cost?
- Does it move strategic KPIs materially?
Power Analysis Inputs
For conversion experiments (two proportions):
- Baseline conversion rate
- MDE (absolute points or relative uplift)
- Alpha (e.g., 0.05)
- Power (e.g., 0.8)
Output:
- Required sample size per variant
- Total sample size
- Approximate runtime based on traffic volume