claude-skills-reference/engineering/statistical-analyst/references/statistical-testing-concepts.md

# Statistical Testing Concepts Reference

Deep-dive reference for the Statistical Analyst skill. Keeps SKILL.md lean while preserving the theory.

---

## The Frequentist Framework

All tests in this skill operate in the **frequentist framework**: we define a null hypothesis (H₀) and an alternative (H₁), then ask "how often would we see data this extreme if H₀ were true?"

- **H₀ (null):** No difference exists between control and treatment
- **H₁ (alternative):** A difference exists (two-tailed)
- **p-value:** P(observing this result or more extreme | H₀ is true)
- **α (significance level):** The threshold we set in advance. Reject H₀ if p < α.

### The p-value misconception
A p-value of 0.03 does **not** mean "there is a 97% chance the effect is real."
It means: "If there were no effect, we would see data this extreme only 3% of the time."

---

## Type I and Type II Errors

| | H₀ True | H₀ False |
|---|---|---|
| Reject H₀ | **Type I Error (α)** — False Positive | Correct (Power = 1−β) |
| Fail to reject H₀ | Correct | **Type II Error (β)** — False Negative |

- **α** (false positive rate): Typically 0.05. Reduce it when false positives are costly (medical trials, irreversible changes).
- **β** (false negative rate): Typically 0.20 (power = 80%). Reduce it when missing real effects is costly.

---

## Two-Proportion Z-Test

**When:** Comparing two binary conversion rates (e.g. clicked/not, signed up/not).

**Assumptions:**
- Independent samples
- n×p ≥ 5 and n×(1−p) ≥ 5 for both groups (normal approximation valid)
- No interference between units (SUTVA)

**Formula:**
```
z = (p̂₂ − p̂₁) / √[p̄(1−p̄)(1/n₁ + 1/n₂)]

where p̄ = (x₁ + x₂) / (n₁ + n₂)  (pooled proportion)
```

**Effect size — Cohen's h:**
```
h = 2 arcsin(√p₂) − 2 arcsin(√p₁)
```
The arcsine transformation stabilizes variance across different baseline rates.

---

## Welch's Two-Sample t-Test

**When:** Comparing means of a continuous metric between two groups (revenue, latency, session length).

**Why Welch's (not Student's):**
Welch's t-test does not assume equal variances — it is strictly more general and loses little power when variances are equal. Always prefer it.

**Formula:**
```
t = (x̄₂ − x̄₁) / √(s₁²/n₁ + s₂²/n₂)

Welch–Satterthwaite df:
df = (s₁²/n₁ + s₂²/n₂)² / [(s₁²/n₁)²/(n₁−1) + (s₂²/n₂)²/(n₂−1)]
```

**Effect size — Cohen's d:**
```
d = (x̄₂ − x̄₁) / s_pooled

s_pooled = √[((n₁−1)s₁² + (n₂−1)s₂²) / (n₁+n₂−2)]
```

**Warning for heavy-tailed metrics (revenue, LTV):**
Mean tests are sensitive to outliers. If the distribution has heavy tails, consider:
1. Winsorizing at 99th percentile before testing
2. Log-transforming (if values are positive)
3. Using a non-parametric test (Mann-Whitney U) and flagging for human review

---

## Chi-Square Test

**When:** Comparing categorical distributions (e.g. which plan users selected, which error type occurred).

**Assumptions:**
- Expected count ≥ 5 per cell (otherwise, combine categories or use Fisher's exact)
- Independent observations

**Formula:**
```
χ² = Σ (Oᵢ − Eᵢ)² / Eᵢ

df = k − 1  (goodness-of-fit)
df = (r−1)(c−1)  (contingency table, r rows, c columns)
```

**Effect size — Cramér's V:**
```
V = √[χ² / (n × (min(r,c) − 1))]
```

---

## Wilson Score Interval

The standard confidence interval formula for proportions (`p̂ ± z√(p̂(1−p̂)/n)`) can produce impossible values (< 0 or > 1) for small n or extreme p. The Wilson score interval fixes this:

```
center = (p̂ + z²/2n) / (1 + z²/n)
margin = z/(1+z²/n) × √(p̂(1−p̂)/n + z²/4n²)
CI = [center − margin, center + margin]
```

Always use Wilson (or Clopper-Pearson) for proportions. The normal approximation is a historical artifact.

---

## Sample Size & Power

**Power:** The probability of correctly detecting a real effect of size δ.

```
n = (z_α/2 + z_β)² × (σ₁² + σ₂²) / δ²    [means]
n = (z_α/2 + z_β)² × (p₁(1−p₁) + p₂(1−p₂)) / (p₂−p₁)²    [proportions]
```

**Key levers:**
- Increase n → more power (or detect smaller effects)
- Increase MDE → smaller n (but you might miss smaller real effects)
- Increase α → smaller n (but more false positives)
- Increase power → larger n

**The peeking problem:**
Checking results before the planned end date inflates your effective α. If you peek at 50%, 75%, and 100% of planned n, your true α is ~0.13 instead of 0.05 — a 2.6× inflation of false positives.

**Solutions:**
- Pre-commit to a stopping rule and don't peek
- Use sequential testing (SPRT) if early stopping is required
- Use a Bonferroni-corrected α if you peek at scheduled intervals

---

## Multiple Comparisons

Testing k hypotheses at α = 0.05 gives P(at least one false positive) ≈ 1 − (1 − 0.05)^k

| k tests | P(≥1 false positive) |
|---|---|
| 1 | 5% |
| 3 | 14% |
| 5 | 23% |
| 10 | 40% |
| 20 | 64% |

**Corrections:**
- **Bonferroni:** Use α/k per test. Conservative but simple. Appropriate for independent tests.
- **Benjamini-Hochberg (FDR):** Controls false discovery rate, not family-wise error. Preferred when many tests are expected to be true positives.

---

## SUTVA (Stable Unit Treatment Value Assumption)

A critical assumption for valid A/B tests: the outcome of unit i depends only on its own treatment assignment, not on other units' assignments.

**Violations:**
- Social features (user A sees user B's activity — network spillover)
- Shared inventory (one variant depletes shared stock)
- Two-sided marketplaces (buyers and sellers interact)

**Solutions:**
- Cluster randomization (randomize at the group/geography level)
- Network A/B testing (graph-based splits)
- Holdout-based testing

---

## References

- Imbens, G. & Rubin, D. (2015). *Causal Inference for Statistics, Social, and Biomedical Sciences*. Cambridge.
- Kohavi, R., Tang, D., & Xu, Y. (2020). *Trustworthy Online Controlled Experiments*. Cambridge.
- Cohen, J. (1988). *Statistical Power Analysis for the Behavioral Sciences*. 2nd ed.
- Wilson, E.B. (1927). "Probable Inference, the Law of Succession, and Statistical Inference." *JASA* 22(158): 209–212.