eef020c9e0
Build campaign-analytics, financial-analyst, customer-success-manager, sales-engineer, and revenue-operations skills using the Claude Skills Factory workflow. Each skill includes SKILL.md, Python CLI tools, reference guides, and asset templates. All 16 Python scripts use standard library only with --format json/text support. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
3.4 KiB
3.4 KiB
A/B Test Analysis
Test Name: [Descriptive test name] Test ID: [Internal tracking ID] Date: [Start Date] - [End Date] Status: [Planning / Running / Complete / Inconclusive]
Hypothesis
If [we change X], then [Y will happen], because [rationale based on data or insight].
Test Design
| Parameter | Detail |
|---|---|
| Variable Tested | [What is being changed] |
| Control (A) | [Description of control variant] |
| Variant (B) | [Description of test variant] |
| Primary Metric | [The main metric being measured] |
| Secondary Metrics | [Additional metrics to monitor] |
| Traffic Split | [50/50, 70/30, etc.] |
| Minimum Sample Size | [Required sample per variant for statistical significance] |
| Minimum Detectable Effect | [Smallest meaningful difference, e.g., 5% lift] |
| Confidence Level | [95% or 99%] |
| Expected Duration | [X days/weeks based on traffic and sample size] |
Targeting
| Criterion | Value |
|---|---|
| Audience | [Who sees the test] |
| Channel | [Where the test runs] |
| Device | [All / Desktop / Mobile] |
| Geography | [Regions included] |
| Exclusions | [Who is excluded and why] |
Results
Primary Metric: [Metric Name]
| Variant | Sample Size | Conversions | Rate | Lift vs Control |
|---|---|---|---|---|
| Control (A) | % | - | ||
| Variant (B) | % | % |
Statistical Significance: [Yes/No] at [X]% confidence P-value: [X.XXX]
Secondary Metrics
| Metric | Control (A) | Variant (B) | Lift | Significant? |
|---|---|---|---|---|
| [Metric 1] | % | [Yes/No] | ||
| [Metric 2] | % | [Yes/No] | ||
| [Metric 3] | % | [Yes/No] |
Segment Analysis
| Segment | Control Rate | Variant Rate | Lift | Notes |
|---|---|---|---|---|
| Desktop | % | % | % | |
| Mobile | % | % | % | |
| New Visitors | % | % | % | |
| Returning Visitors | % | % | % | |
| [Custom Segment] | % | % | % |
Revenue Impact Estimate
| Metric | Value |
|---|---|
| Projected Annual Lift | [X]% |
| Projected Additional Revenue | $[X] |
| Projected Additional Conversions | [X] |
| Confidence in Estimate | [High/Medium/Low] |
Decision
Winner: [Control / Variant / Inconclusive]
Rationale: [Why this decision was made, citing specific metrics and statistical significance]
Implementation Plan:
- [Step 1: e.g., Roll out variant to 100% of traffic]
- [Step 2: e.g., Update creative assets across campaigns]
- [Step 3: e.g., Monitor for X days post-implementation]
- [Step 4: e.g., Document learnings in knowledge base]
Learnings
What we learned:
- [Key learning 1]
- [Key learning 2]
- [Key learning 3]
Follow-up tests to consider:
- [Next test idea based on results]
- [Next test idea based on results]
Quality Checks
- Sample size reached minimum threshold
- Test ran for at least 1 full business cycle (7 days minimum)
- No external factors (holidays, outages, promotions) affected results
- Segments were balanced between variants
- No sample ratio mismatch (SRM) detected
- Results reviewed by at least 2 team members
Template from campaign-analytics skill. Statistical significance calculations require external tools (e.g., online calculators or scipy).