Experiment Designer

What this skill helps you do

Plan and evaluate A/B and multivariate product experiments with defensible statistical decisions. Provides a structured workflow from hypothesis writing in If/Then/Because format through sample size estimation, ICE prioritization scoring, and results interpretation with practical significance thresholds. Includes a Python sample size calculator script, a hypothesis quality checklist, common experiment pitfall warnings (underpowered tests, peeking, instrumentation drift), and statistical guardrails that emphasize business significance over p-value hunting.

Install and get started

Copy the full instructions into your AI tool. Test one low-risk example before connecting real store data.

Original Skill instructions

Design, prioritize, and evaluate product experiments. Follow this workflow: 1) Write hypothesis in If/Then/Because format — If [intervention], Then [metric] changes by [direction/magnitude], Because [mechanism]. 2) Define primary, guardrail, and secondary metrics before testing. 3) Estimate sample size via scripts/sample_size_calculator.py using baseline rate, MDE, alpha, and power. 4) Prioritize with ICE score (Impact × Confidence × Ease / 10). 5) Launch with pre-defined stopping rules; avoid repeated peeking. 6) Interpret results using confidence intervals and practical significance, not just p-values. Guard against underpowered tests, mid-test changes, sample ratio mismatch, and instrumentation drift.

Useful tasks

• Planning A/B tests with proper sample size estimation and statistical power
• Writing testable hypotheses with clear success and failure criteria
• Prioritizing experiment backlogs using ICE scoring framework
• Interpreting experiment results with confidence intervals and business context
• Setting up guardrail metrics to protect against unintended negative effects

How to use it

• Always define a single primary metric before launching — multiple primary metrics lead to cherry-picking and inflated false positives
• Use absolute MDE for binary metrics (conversion rates) and relative MDE for continuous metrics (revenue, time)
• A statistically significant result with a tiny effect size may not be worth shipping — compare against your practical significance threshold
• Never stop an experiment early just because results look good — random fluctuations can produce misleading winner signals
• Pre-register your hypothesis, metrics, and stopping rules; changing them mid-test invalidates your statistical framework

More skills for this workflow

• Email Campaign Writer

  Write a month of ecommerce emails in 30 minutes

• Brand Voice Content Engine

  Generate all marketing content in your exact brand voice

• Ad Creative

  Generate and scale high-converting ad copy across Google, Meta, LinkedIn, and more

• Launch Strategy

  Plan product launches that build momentum and convert interest into users