Why A/B Testing Concepts Are Non-Negotiable for Data Analysts Today

Every product decision at a data-driven company goes through experimentation. When Swiggy tests a new checkout flow, when Paytm tweaks its cashback banner, or when Flipkart experiments with a new recommendation algorithm — they rely on A/B testing to make confident, statistically valid decisions. As a data analyst, you are the person in the room who designs these experiments, monitors them in real time, and interprets the results for product managers and business leaders.

A/B testing, at its core, is the process of splitting your user base into two or more groups — a control group (A) and one or more treatment groups (B, C, etc.) — exposing each group to a different version of a product feature, UI element, or algorithm, and then using statistical methods to determine whether the observed difference in key metrics is real or just due to random chance.

But here’s the thing interviewers at top Indian and global tech companies have started doing: they no longer ask “What is A/B testing?” They now ask nuanced questions like “How would you handle a situation where your A/B test shows significance after just two days?” or “What is novelty effect and how do you control for it?” These questions separate candidates who have just read a Wikipedia definition from those who have actually run experiments in production environments.

The concepts you need to master include: null and alternative hypotheses, Type I and Type II errors, statistical power, p-values, confidence intervals, minimum detectable effect (MDE), sample size calculation, test duration, metric selection, guardrail metrics, network effects, and the Bonferroni correction for multiple testing. If even two or three of those terms felt unfamiliar, keep reading — this post breaks them all down in plain language with interview context baked in.

A/B Testing Interview Questions Top Companies Are Asking Right Now

Based on recent interview experiences shared by candidates who cracked roles at Swiggy, Meesho, Razorpay, Amazon India, and Google, here are the A/B testing questions that are trending in 2026 interview cycles. These questions appear across product analytics, business analyst, and data science analyst roles. Prepare a structured STAR-style answer for each of these, backed with real numbers and statistical reasoning.

1. “You run an A/B test and the result shows a p-value of 0.03. Your product manager wants to ship immediately. What questions do you ask before recommending a decision?” — This tests whether you blindly trust p-values or understand factors like test duration, sample size adequacy, novelty effects, metric trade-offs, and guardrail metric health. A great answer covers all of these systematically.
2. “How would you calculate the sample size needed for an A/B test on Paytm’s UPI homepage, where the baseline conversion rate is 12% and you want to detect a 1.5% absolute lift with 80% power at a 5% significance level?” — This is a quantitative question that tests your knowledge of the sample size formula: n = (Z_α/2 + Z_β)² × [p1(1−p1) + p2(1−p2)] / (p1−p2)². Candidates are expected to either recall this formula or describe the logic clearly.
3. “Swiggy runs an A/B test on a new delivery ETA feature. After two weeks, the test group shows higher orders but also higher cancellations. How do you interpret and present this result to the product team?” — This tests your understanding of metric trade-offs, guardrail metrics (cancellation rate acting as a guardrail), and your ability to tell a coherent data story rather than just reporting numbers.
4. “What is the difference between a one-tailed and two-tailed test, and when would you use each in a product experimentation setting?” — A classic conceptual question. One-tailed tests are appropriate when you only care about improvement (e.g., “does this increase revenue?”), while two-tailed tests are used when any directional change matters (e.g., “does this change engagement at all?”).
5. “Your A/B test has been running for 10 days and you’re at 95% confidence. Your business stakeholder wants to stop early. What is p-hacking and how do you explain the risk to a non-technical audience?” — This tests your communication skills and your understanding of the multiple comparisons problem and optional stopping bias.

Python Code: Running an A/B Test from Scratch Like a Pro

In many analytics interviews — especially at startups like Razorpay, CRED, and Zepto — you will be given a dataset and asked to run an A/B test end to end in Python. You need to be comfortable with calculating conversion rates, running a two-proportion z-test, interpreting the p-value, and visualizing the confidence intervals. Here is a clean, interview-ready Python snippet that covers all of this. Practice running this on a sample dataset before your interview.

import numpy as np
from scipy import stats
import matplotlib.pyplot as plt

# --- Sample Data (mimicking a Swiggy checkout A/B test) ---
# Control Group (A): Old checkout flow
control_conversions = 1200   # users who completed order
control_total = 10000        # total users in control

# Treatment Group (B): New one-click checkout
treatment_conversions = 1350
treatment_total = 10000

# --- Step 1: Calculate Conversion Rates ---
p_control = control_conversions / control_total
p_treatment = treatment_conversions / treatment_total

print(f"Control Conversion Rate:   {p_control:.4f} ({p_control*100:.2f}%)")
print(f"Treatment Conversion Rate: {p_treatment:.4f} ({p_treatment*100:.2f}%)")
print(f"Absolute Lift:             {(p_treatment - p_control)*100:.2f}%")
print(f"Relative Lift:             {((p_treatment - p_control)/p_control)*100:.2f}%")

# --- Step 2: Two-Proportion Z-Test ---
count = np.array([treatment_conversions, control_conversions])
nobs = np.array([treatment_total, control_total])

from statsmodels.stats.proportion import proportions_ztest
z_stat, p_value = proportions_ztest(count, nobs, alternative='larger')

print(f"\nZ-Statistic: {z_stat:.4f}")
print(f"P-Value:     {p_value:.4f}")

# --- Step 3: Interpret Results ---
alpha = 0.05
if p_value < alpha:
    print(f"\n✅ Result: Statistically significant at {alpha*100}% significance level.")
    print("   Recommend shipping the new checkout flow.")
else:
    print(f"\n❌ Result: NOT statistically significant at {alpha*100}% significance level.")
    print("   Do not ship. Consider extending the test or revising the feature.")

# --- Step 4: Calculate 95% Confidence Interval for Lift ---
se = np.sqrt(
    (p_control * (1 - p_control) / control_total) +
    (p_treatment * (1 - p_treatment) / treatment_total)
)
lift = p_treatment - p_control
ci_lower = lift - 1.96 * se
ci_upper = lift + 1.96 * se

print(f"\n95% Confidence Interval for Lift: [{ci_lower*100:.3f}%, {ci_upper*100:.3f}%]")
print("If this interval does not include 0, the result is statistically significant.")