39 lines
1.3 KiB
Markdown
39 lines
1.3 KiB
Markdown
---
|
||
name: ab-testing
|
||
description: Use when designing experiments for subject lines, offers, cadences, or
|
||
journeys.
|
||
---
|
||
|
||
# Experimentation & A/B Testing Skill
|
||
|
||
## When to Use
|
||
- Validating new subject lines or creative.
|
||
- Testing segmentation hypotheses (persona vs behavior).
|
||
- Optimizing cadence, timing, or automation triggers.
|
||
|
||
## Framework
|
||
1. **Hypothesis** – define expected uplift + rationale.
|
||
2. **Metric Selection** – primary (open/click/conv) + guardrails (unsubs, spam).
|
||
3. **Sample Sizing** – ensure stat significance (min 500 recipients per variant or use power calculator).
|
||
4. **Execution** – randomize, keep variants isolated, limit simultaneous tests.
|
||
5. **Analysis** – use z-test or Bayesian uplift; document learnings.
|
||
|
||
## Templates
|
||
- Experiment brief (hypothesis, segments, KPI, risk guardrails).
|
||
- Variant table (control vs test inputs, creative asset links, owner).
|
||
- Calculator sheet for minimum detectable effect + sample size.
|
||
- Post-test debrief doc capturing learnings + rollout plan.
|
||
|
||
## Experiment Ideas
|
||
- Subject line vs preview text combos.
|
||
- CTA placement (hero vs footer).
|
||
- Personalization depth (basic vs dynamic modules).
|
||
- Wait times between touches.
|
||
|
||
## Tips
|
||
- Run no more than two tests per journey simultaneously.
|
||
- Recycle learnings into playbooks + automation templates.
|
||
- Segment results by persona to catch hidden signals.
|
||
|
||
---
|