How to use the A/B Test Significance Calculator
- 1
Enter visitors and conversions for the control (A) and the variant (B).
- 2
Read the confidence level and lift.
- 3
Only call a winner at 95% confidence or higher, and after a full business cycle.
How significance is calculated
The calculator runs a two-proportion z-test. It compares the two conversion rates and estimates how likely a difference this large would be if the versions actually performed the same. Confidence is 1 minus that probability (the p-value).
Avoid the common traps
Most "winning" tests that later disappoint broke one of these rules:
- Decide the sample size before you start, and don't stop early when it looks good
- Run at least one full week to cover weekday and weekend behavior
- Test one meaningful change at a time
- Re-check the winner after launch: novelty effects fade
Frequently asked questions
What confidence level should I use?
95% is the standard. For low-risk changes some teams accept 90%; for big decisions, wait for 99%.
How many visitors do I need?
It depends on your baseline rate and the lift you want to detect. Small lifts on low conversion rates need thousands of visitors per version.
Is this one-tailed or two-tailed?
Two-tailed, so it checks whether B is better or worse than A.
Is my data stored?
No. Everything is calculated in your browser.