Frequently Asked Questions about A/B Split Test Calculations
You need to enter four main values:
- Control Visitors (A): Total number of users who saw version A
- Control Conversions (A): Number of users who converted in version A
- Variation Visitors (B): Total number of users who saw version B
- Variation Conversions (B): Number of users who converted in version B
The calculator offers two key settings:
Test Type:
One-sided: Tests if variant B is better than variant A
Two-sided: Tests if variant B is different from variant A (either better or worse)
Confidence Level: Choose between 90%, 95% (standard), or 99%
The calculator shows:
- A clear success/failure message
- Conversion rates for both variants
- Relative difference between variants
- Statistical power percentagep-value of the test
Conversion rate is calculated as:
Control rate = Control Conversions / Control Visitors
Variation rate = Variation Conversions / Variation Visitors
All rates are displayed as percentages in the results.
The p-value shows the probability of seeing these results by chance if there was no real difference between variants. Lower p-values indicate stronger evidence of a real difference:
- Below 0.05 for 95% confidence
- Below 0.01 for 99% confidence
- Below 0.10 for 90% confidence
Your test is significant if the p-value is less than (1 - your confidence level).
Ready to get more customers for your SaaS?
Book in a call and I’ll show you how I’d scale your acquisition with Google Ads for your product.