Skip to content This library is free, offered as dana. If it helps your work, donate here →
penguin tree ai
← back to libraryDonate here

This is one of 3,622 free prompts in the penguin tree ai library. Sign up once and the whole library opens, with new prompts by email.

open the library →

A/B test analyzer

Analyzes A/B test results from sample sizes, conversion rates, and significance metrics — calculating statistical significance, confidence intervals, lift, minimum detectable effect, and practical significance, then…

download .md

Purpose

Turn raw experiment numbers into a defensible decision — calculating statistical significance, confidence intervals, and observed lift, then weighing them against practical significance to recommend whether to ship, iterate, or kill the variant. Eliminates the two most common analysis failures: declaring a winner on an underpowered test and killing a directionally promising variant that never reached significance.

When to Use

Invoke this skill when the user asks to:

  • Interpret the results of a completed A/B or split test
  • Determine whether a variant reached statistical significance
  • Decide whether to ship, iterate on, or kill a test variant
  • Check whether a test was adequately powered or ran long enough
  • Diagnose an inconclusive, flat, or surprising experiment result
  • Calculate lift, confidence interval, or required sample size for a conversion test

Inputs Required

Before starting, collect:

  • Variant structure: Number of variants and which is control (A) vs. treatment(s) (B, C…)
  • Sample size per variant: Visitors/users exposed in each arm
  • Conversions per variant: Conversion count or conversion rate for each arm
  • Primary metric: What "conversion" means here (signup, purchase, click, activation, etc.)
  • Significance threshold: Alpha level — default 0.05 (95% confidence) if not provided
  • Minimum effect that matters: Smallest lift worth shipping for (business-defined), if known
  • Test duration & context: How long it ran, any known confounds (seasonality, bugs, audience changes)

If the user provides only conversion rates, derive conversion counts from sample sizes. If the minimum effect that matters is missing, infer a reasonable threshold from the metric type and flag the assumption.

Instructions

Step 1: Normalize and Validate the Inputs

  • Compute conversion rate per arm: conversions ÷ sample size.
  • Confirm arms are comparable in size — flag any allocation imbalance over 60/40, which can signal SRM (sample ratio mismatch) and contaminate results.
  • Flag any arm with fewer than ~100 conversions as low-conversion-count risk — significance math becomes unreliable below this.
  • Restate the inputs back cleanly in a small table so the user can confirm before interpretation.

Step 2: Calculate Statistical Significance

For a two-proportion comparison (control vs. each treatment):

  • Compute the pooled proportion and standard error of the difference.
  • Compute the z-score: (p_treatment − p_control) ÷ standard error.
  • Convert to a two-tailed p-value.
  • Compare to alpha. State plainly: significant or not significant at the chosen threshold.
  • Show the work in one line so the result is auditable — do not just assert a p-value.

Step 3: Calculate Lift and Confidence Interval

  • Relative lift: (p_treatment − p_control) ÷ p_control, as a percentage.
  • Absolute lift: p_treatment − p_control, in percentage points.
  • 95% confidence interval on the absolute difference: difference ± 1.96 × standard error.
  • Interpret the interval explicitly: if it crosses zero, the direction of effect is uncertain even if the point estimate looks positive.

Step 4: Assess Statistical Power and Adequacy

  • Estimate the minimum detectable effect (MDE) the test could reliably catch given the sample sizes at 80% power.
  • If the observed effect is smaller than the MDE, label the test underpowered — a "not significant" result here means inconclusive, not "no difference."
  • Note whether the test ran at least one full business cycle (typically 7+ days) to absorb day-of-week effects.

Step 5: Separate Statistical from Practical Significance

  • A result can be statistically significant but commercially trivial, or directionally strong but underpowered.
  • Compare the observed lift (and its confidence interval lower bound) against the minimum effect that matters.
  • Classify into one of four states:
  • Significant + meaningful → ship
  • Significant + trivial → ship only if cost-free; otherwise deprioritize
  • Not significant + underpowered → iterate / extend / rerun with more traffic
  • Not significant + well-powered, flat → kill the variant

Step 6: Flag Validity Threats

Check for and call out anything that would invalidate the read:

  • Sample ratio mismatch (allocation skew)
  • Peeking / early stopping inflating false positives
  • Novelty or primacy effects in short tests
  • Known confounds from the inputs (bug windows, marketing spikes, seasonality)

Step 7: Deliver the Recommendation

Give one clear verdict — Ship / Iterate / Kill — with a 2–3 sentence rationale tied to the specific numbers, followed by the single most important caveat the user should account for before acting.

Output Format

  • Structure: Inputs recap table → calculated results → interpretation → recommendation
  • Length: 500–800 words
  • Format: Mixed — a results table for the numbers, prose for interpretation, a bolded one-line verdict
  • Sections: Inputs Recap → Significance & Lift Results → Power Assessment → Statistical vs. Practical Significance → Validity Threats → Recommendation
  • Tone: Senior experimentation analyst — precise, numerate, willing to say "inconclusive" rather than overclaim a winner

Quality Checklist

Before delivering output, verify:

  • [ ] Conversion rates, p-value, lift, and confidence interval are all calculated and shown — not asserted
  • [ ] The confidence interval is interpreted relative to zero (does it cross zero or not)
  • [ ] Statistical significance and practical significance are addressed as separate questions
  • [ ] An underpowered "null" result is labeled inconclusive, not "no effect"
  • [ ] At least one validity threat (SRM, peeking, novelty, confound) is explicitly checked
  • [ ] A single Ship/Iterate/Kill verdict is stated with a numbers-based rationale and one key caveat

Notes

  • The most common error this skill prevents is treating "p > 0.05" as proof of no difference — on an underpowered test, that's an absence of evidence, not evidence of absence.
  • When multiple treatments are compared against one control, note that running several comparisons inflates false-positive risk; mention a Bonferroni-style correction if there are 3+ variants.
  • Bayesian framing (probability that B beats A, expected loss) is often more decision-useful than p-values for product teams — offer it as an alternative read when the user seems uncomfortable with frequentist thresholds.
  • A point estimate without its confidence interval is the single most misleading number in experimentation; always pair them.

How to use this skill

This file is a Claude Code skill in Anthropic's standard format — markdown with YAML frontmatter at the top.

Installing in Claude Code:

  • ~/.claude/skills/<folder-name>/SKILL.md — auto-loads in every session anywhere on your machine (recommended for skills you'll use across multiple projects)
  • <your-project>/.claude/skills/<folder-name>/SKILL.md — auto-loads only when working in that project (recommended for project-specific skills)

Restart Claude Code or open a new session to register.

Using in Claude.ai, ChatGPT, Gemini, or other AI tools:

  1. Drag this file into a new chat, or copy and paste its contents
  2. Send your task or request as your next message — the AI will apply the skill where it fits

For Claude Code skill details: https://docs.anthropic.com/en/docs/claude-code/skills