A/B Testing Complete Guide for Data Analysts — With Python 2026 | Data Analyst Interview
🧪 A/B Testing Guide

A/B Testing — Complete Guide for Data Analysts (With Python)

Everything you need to know about A/B testing for data analyst interviews — hypothesis design, statistical significance, sample size calculation, and common pitfalls.

Book Free Session ← Analytics Round
Core Concepts
A/B testing from first principles

You don’t need to be a statistician — but you must understand these six concepts to answer any A/B testing question in an interview.

💡

Hypothesis

Every test starts with a falsifiable hypothesis. Format: “We believe [change] will increase [metric] for [users] because [reason].”

  • Null hypothesis (H₀): no effect
  • Alternative (H₁): there is an effect
  • One-tailed vs two-tailed tests
🎯

Primary & Guardrail Metrics

Define what you’re optimising (primary) and what must not break (guardrail) before running the experiment.

  • Primary: conversion rate, revenue
  • Guardrail: session length, NPS
  • Pre-commit — no changing after
📏

Sample Size

Calculate before you run. Too small = underpowered test. Too long = wasted time. Use a power calculator.

  • Baseline conversion rate
  • Minimum detectable effect (MDE)
  • 80% power, 95% significance
⚡

Statistical Significance

p-value < 0.05 means less than 5% chance of seeing this result if the null hypothesis were true. Not "95% the test won."

  • p-value interpretation
  • Confidence intervals
  • Two-sample z-test vs t-test
⚠️

Type I & Type II Errors

Type I: false positive — you ship a feature that doesn’t work. Type II: false negative — you kill a feature that did work.

  • α = Type I error rate (usually 0.05)
  • β = Type II error rate (usually 0.20)
  • Power = 1 − β
🚫

Common Pitfalls

Peeking, novelty effect, network effects, leakage — the traps that invalidate real experiments.

  • Peeking = inflated Type I error
  • Novelty effect fades in 2–3 weeks
  • Network leakage in social products
Python Implementation
Two-proportion z-test for conversion rate

The most common A/B test implementation for conversion rate experiments — run this in a Python mock and explain every line.

from scipy import stats
import numpy as np
 
# Control group
control_users = 10000
control_conversions = 500 # 5% baseline
 
# Treatment group
treat_users = 10000
treat_conversions = 580 # 5.8% — is this real?
 
p_ctrl = control_conversions / control_users
p_treat = treat_conversions / treat_users
 
# Pooled proportion under H₀
p_pool = (control_conversions + treat_conversions) / (control_users + treat_users)
se = np.sqrt(p_pool * (1 – p_pool) * (1/control_users + 1/treat_users))
 
z_stat = (p_treat – p_ctrl) / se
p_value = 2 * (1 – stats.norm.cdf(abs(z_stat)))
 
print(f”Control: {p_ctrl:.1%}, Treatment: {p_treat:.1%}”)
print(f”Z-stat: {z_stat:.2f}, p-value: {p_value:.4f}”)
print(“Significant” if p_value < 0.05 else “Not significant”)
Interview Questions
Real A/B testing questions from 2026 interviews

These are the questions candidates get wrong most often — with the model answers.

What does p < 0.05 actually mean?
It means: if the null hypothesis (no effect) were true, there would be less than a 5% probability of observing a result as extreme as ours just by chance. It does NOT mean “there is a 95% chance the treatment works.” This is the most common misinterpretation — stating it correctly in an interview immediately marks you as statistically literate.
Your test ran for 3 days and shows p = 0.03 with a 15% lift. Do you ship?
No — for three reasons. (1) Peeking: you likely checked before the pre-committed end date, inflating Type I error. (2) Novelty effect: a 15% lift on day 3 often decays to 2–3% by day 21 as users adjust to the new experience. (3) Sample size: 3 days is usually insufficient for adequate statistical power. Run to the pre-defined end date, check guardrail metrics, then decide.
How do you handle a test where you can’t randomise at the user level?
Use cluster randomisation — randomise at the level of city, store, or time period instead of individual users. This prevents leakage (one user in control seeing the treatment via their network). The trade-off is lower statistical power because you have fewer randomisation units. You need a larger sample of clusters to compensate. For time-based tests, use switchback experiments — alternate treatment and control periods.

Want to practise A/B testing questions?

Book a free analytics mock — we cover experiment design, statistical interpretation, and Python implementation with live feedback.

Book Free Analytics Mock
Subscribe
Subscribe