Question 1

How does this A/B test significance calculator work?

Accepted Answer

It runs a pooled two-proportion z-test. It compares the conversion rates of A and B, measures how big the gap is relative to the noise you'd expect at those sample sizes, and converts that into a p-value. If the p-value is below 1 minus your confidence level (0.05 at 95%), the result is statistically significant.

Question 2

What does the p-value mean?

Accepted Answer

The p-value is the probability of seeing a difference at least this large if A and B actually convert at the same rate. A p-value of 0.03 means a gap this big would show up about 3% of the time from random chance alone. It is not the probability that B is better.

Question 3

Should I use a one-sided or two-sided test?

Accepted Answer

Use two-sided unless you decided before the test that you only care whether B beats A. Two-sided is stricter and catches the case where your change made things worse. One-sided tests reach significance faster but can't tell you B lost.

Question 4

How is sample size calculated?

Accepted Answer

Sample size per variant uses your baseline conversion rate, the minimum detectable effect (the smallest relative lift you care about), the confidence level, and statistical power. Smaller effects and lower baseline rates need far more visitors. Detecting a 10% lift on a 3% baseline takes tens of thousands of visitors per variant.

Question 5

How long should I run an A/B test?

Accepted Answer

Until each variant reaches the sample size you calculated up front, and for at least one full week so weekday and weekend behavior are both represented. Stopping as soon as the result looks significant (peeking) sharply increases false positives.

Question 6

Can I test more than two variants?

Accepted Answer

Yes, and the sample size calculator scales the total visitors for up to four variants. Comparing many variants against control increases the odds of a false positive, so consider a stricter confidence level (like 99%) when you run several at once.

Question 7

What if my site doesn't have enough traffic to A/B test?

Accepted Answer

Test bigger changes (a new headline or offer, not a button color) so the effect is large enough to detect, test on your highest-traffic page, or measure a higher-volume step such as clicks to signup instead of paid conversions.

A/B test calculator

Test results

Statistical significance, without the stats degree

Significance: is the difference real?

Sample size: how much traffic you need

Common A/B testing mistakes

What to test first

Let us run the tests, on our dime if they lose

FAQs

Not sure if Conversion Factory is the right fit?