A/B Test Significance
Checks whether a difference between two versions is likely to be real, not chance.
z = (p_B − p_A) ÷ √(p(1 − p)(1/n_A + 1/n_B))
Worked example: p_A = 0.10, p_B = 0.13 and pooled p = 0.115, so z ≈ 2.1.
Why it matters
Without it, teams ship random noise and call it a win.
How to use it
- Decide the sample size before you start. A rough guide per version is n ≈ 16 × p(1 − p) ÷ d², where d is the smallest lift you care about.
- Do not stop early when it looks good.
- A z-score above 1.96 in size is significant at the 95% level.
Example
100 of 1,000 convert on A and 130 of 1,000 on B, giving z ≈ 2.1, which is significant.
Watch out
A significant result is not always a meaningful one. A tiny lift can be real and still not be worth shipping.