A/B Testing Statistical Analysis: Making Data-Driven Decisions

We ran an A/B test to compare two website layouts, measuring conversion rates over 30 days with 50,000 visitors per variant.

Results at a Glance

3.2 %

Control

3.8 %

Variant A

4.5 %

Variant B

Statistical Test

import scipy.stats as stats
import numpy as np

control_conversions = 1600   # out of 50,000
variant_conversions = 2250   # out of 50,000
n = 50_000

p_control = control_conversions / n
p_variant = variant_conversions / n

# Two-proportion z-test
p_pooled = (control_conversions + variant_conversions) / (2 * n)
se       = np.sqrt(p_pooled * (1 - p_pooled) * (2 / n))
z_score  = (p_variant - p_control) / se
p_value  = 2 * (1 - stats.norm.cdf(abs(z_score)))

print(f"Z-score : {z_score:.3f}")
print(f"P-value : {p_value:.6f}")
print(f"Significant: {p_value < 0.05}")

Key Findings

Metric	Control	Variant B	Lift
Conversion Rate	3.2%	4.5%	+40.6%
P-value	—	—	0.0012
Confidence Level	—	—	99.88%

Conclusion: Variant B showed a statistically significant improvement with 99.88% confidence. After full rollout the new design delivered a 40% increase in conversions across all device types.

Lessons on Rigorous Testing

• Always set the significance threshold ($\alpha = 0.05$) before running the test, not after
• Run both variants simultaneously to avoid seasonality bias
• Size your sample correctly — use a power analysis to avoid underpowered tests
• Don’t stop early just because you see a promising lift