AB Test Calculator

An AB test compares two sample groups (Control and Treatment) to see if a specific change impacts user behavior. First, conversion proportions are calculated as p1 = t1 / n1 and p2 = t2 / n2, where t represents successful conversions and n represents total sample sizes. To test for statistical significance, the tool calculates a pooled proportion p = (t1 + t2) / (n1 + n2) and evaluates the Z-score using the formula: z = (p1 - p2) / sqrt(p * (1 - p) * (1/n1 + 1/n2)). This Z-score reflects how many standard deviations away from zero your observed difference lies, letting you evaluate comparative error against your chosen confidence level.

Imagine you run an e-commerce checkout experiment. In Control (Group 1), 150 out of 2,000 visitors complete a purchase (n1 = 2000, t1 = 150), yielding a conversion rate p1 of 7.5%. In the Treatment (Group 2), with a redesigned button, 190 out of 2,000 visitors convert (n2 = 2000, t2 = 190), yielding a conversion rate p2 of 9.5%. The absolute difference is 2.0%. The pooled proportion becomes p = (150 + 190) / (2000 + 2000) = 340 / 4000 = 0.085. Plugging these figures into the Z-score formula results in z = (0.075 - 0.095) / sqrt(0.085 * 0.915 * (1/2000 + 1/2000)) = -0.02 / sqrt(0.085 * 0.915 * 0.001) = -0.02 / sqrt(0.000077775) = -0.02 / 0.0088 = -2.27. Since an absolute Z-score of 2.27 exceeds the standard 1.96 threshold for a 95% confidence level, you can safely conclude the variation's success is statistically significant.

Always pre-determine your required sample size before launching your experiment to avoid peeking and false-positive results. Ensure your traffic is split randomly between groups to eliminate selection bias. Finally, run your test for at least one full business cycle (such as a full week) to account for day-of-the-week behavioral variances.