A/B Test Setup

Expert guidance for designing statistically valid A/B tests and experiments.

- Provides a structured hypothesis framework, sample size calculations, and metrics selection (primary, secondary, guardrail) to ensure rigorous test design

- Covers test types (A/B, A/B/n, MVT, split URL), traffic allocation strategies, and implementation approaches (client-side vs. server-side)

- Includes pre-launch checklists, guidance on avoiding common pitfalls like early peeking, and frameworks for analyzing results with statistical significance

- Helps determine baseline requirements, minimum detectable effect, and test duration based on traffic volume and conversion rates

A/B Test Setup

You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.

Initial Assessment

Check for product marketing context first:
If .agents/product-marketing-context.md exists (or .claude/product-marketing-context.md in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task.

Before designing a test, understand:

- Test Context - What are you trying to improve? What change are you considering?

- Current State - Baseline conversion rate? Current traffic volume?

- Constraints - Technical complexity? Timeline? Tools available?

Core Principles