A/B Test Significance Calculator
Check if your A/B test results are significant
How to use
Enter the number of visitors and conversions for both your control (A) and variation (B). Choose your confidence level (95% is standard). The calculator tells you if the difference is statistically significant, the lift percentage, and confidence intervals. Switch to sample size mode to plan how much traffic you need before running a test.
Formula
Examples
Landing page headline test
Control: 1,000 visitors, 100 conversions (10%). Variation: 1,000 visitors, 130 conversions (13%). The 30% relative lift is statistically significant at 95% confidence (p = 0.032). You can confidently deploy the new headline.
Inconclusive test
Control: 500 visitors, 25 conversions (5%). Variation: 500 visitors, 30 conversions (6%). Despite a 20% lift, the result is not significant (p = 0.47) because the sample size is too small. You need approximately 5,000 visitors per group to detect this effect size reliably.
Planning sample size
Your current conversion rate is 3%. You want to detect at least a 20% relative improvement. At 95% confidence and 80% power, you need about 12,000 visitors per group, or 24,000 total. At 1,000 daily visitors, the test should run about 24 days.
Frequently asked questions
What does "statistically significant" mean?
Statistical significance means the observed difference is unlikely to have occurred by random chance alone. At 95% confidence, a significant result means there is less than a 5% probability that the difference is due to randomness. It does not guarantee the effect is large or practically meaningful.
What confidence level should I use?
95% is the industry standard for most A/B tests. Use 99% when the cost of a wrong decision is high (pricing changes, major redesigns). Use 90% for lower-stakes tests where speed matters more than certainty. Never go below 90%.
How long should I run my A/B test?
Use the sample size calculator to determine how many visitors you need, then divide by your daily traffic. Always run tests for at least one full business cycle (typically 1-2 weeks) to account for day-of-week effects. Never stop a test early just because it looks significant.
What is the p-value?
The p-value is the probability of observing a difference as large as (or larger than) the one you measured, assuming there is no real difference between the variations. A p-value of 0.03 means there is a 3% chance the result is due to randomness. Lower p-values indicate stronger evidence.
Can I test more than two variations?
This calculator handles two-variation (A/B) tests. For tests with three or more variations (A/B/C), you need to adjust for multiple comparisons (Bonferroni correction or similar). Run separate A/B comparisons and apply a stricter significance threshold: divide 0.05 by the number of comparisons.
About this tool
Check if your A/B test results are statistically significant. Enter visitors and conversions for each variation to get p-value, lift, confidence intervals, and sample size.
All calculations are performed locally in your browser. Your data never leaves your device.