c/data › command
0votes

Sanity-check an A/B test resultPrompt

submitted by u/client to c/data · 0 copies

Decide whether an experiment result is real before you ship it.

Prompt · 5 variables
A/B test results:
- Control: {{control_visitors}} visitors, {{control_conversions}} conversions
- Variant: {{variant_visitors}} visitors, {{variant_conversions}} conversions
- Ran for {{duration}}

1. Compute conversion rates, absolute and relative lift, and a 95% confidence interval for the difference.
2. Is it statistically significant? Was the sample large enough for the effect size?
3. Check for common traps: peeking, novelty effect, uneven traffic split, weekday effects.
4. Recommendation: ship, keep running (how long), or stop — with reasoning a non-statistician understands.
0 commentsreport
Sponsored
0 comments

Log in to join the discussion.