Why A/B Testing Is a Core Skill Every Data Analyst Must Master Right Now

Think about what Swiggy does every single day — they test new UI layouts, discount messaging, delivery time displays, and even the colour of their “Order Now” button. Paytm runs hundreds of experiments monthly to improve conversion on their payments funnel. Flipkart uses A/B testing to decide which product recommendation algorithm drives more GMV. In every single one of these scenarios, a data analyst is sitting at the centre of the experiment — designing it, monitoring it, and drawing conclusions from it.

A/B testing, at its core, is the practice of comparing two versions of something (version A vs version B) to determine which performs better against a defined metric. But here’s what most candidates get wrong — they think it’s just about running the test. The real skill is in understanding the statistics behind the test, knowing when results are trustworthy, recognising when an experiment has been poorly designed, and communicating findings to non-technical stakeholders clearly.

In Indian tech companies, product and growth teams rely heavily on experimentation frameworks. Tools like Optimizely, in-house platforms at Flipkart (called Experimentation Hub), and even basic Google Optimize setups are commonly used. As a data analyst, you’ll be expected to own the metrics definition, sample size calculation, statistical significance testing, and post-experiment analysis. Interviewers at top companies are increasingly asking candidates to walk through an entire A/B test lifecycle — from hypothesis to decision — in case study rounds. If you can’t do that confidently, you’re leaving offers on the table.

The good news? The fundamentals are learnable and repeatable. Let’s get into exactly what you need to know.

A/B Testing Interview Questions That Top Indian Tech Companies Are Asking Right Now

Companies like Google India, Flipkart, PhonePe, and Zomato are asking increasingly sophisticated A/B testing questions in their data analyst interviews. These questions test not just theoretical knowledge but your ability to apply concepts to real product scenarios. Here are the types of questions you should be preparing for — along with what a strong answer looks like at a high level.

1. “How would you design an A/B test to evaluate whether a new recommendation algorithm increases average order value on Flipkart’s homepage? Walk me through every step.” — This tests your end-to-end knowledge: hypothesis formation, metric selection (primary vs guardrail metrics), randomisation unit (user-level vs session-level), sample size calculation, test duration, and significance threshold. A strong answer mentions both statistical power and practical significance.
2. “Your A/B test reached statistical significance after just 3 days. Should you stop the test and ship the feature? Why or why not?” — This is a classic trap question. The correct answer is almost always “no” — you need to account for novelty effects, day-of-week bias, and ensure you’ve reached your pre-determined sample size. Interviewers at Swiggy and Paytm love this question because it reveals whether you truly understand the difference between statistical significance and reliable results.
3. “You’re running an A/B test on Swiggy’s discount banner and you notice that the treatment group has a 15% higher click-through rate but a 5% lower conversion rate. How do you interpret and present this finding to the product team?” — This tests your ability to handle conflicting signals, think about the full funnel, and communicate nuanced findings. The best answers discuss metric trade-offs, potential reasons for the divergence (e.g., banner attracting non-intent users), and recommend a follow-up investigation before making a ship/no-ship decision.

Python Skill for A/B Testing: Running a Two-Sample Z-Test and Calculating Statistical Significance

One of the most practical coding tasks you might face in a data analyst interview or take-home assignment is implementing a basic statistical significance test from scratch — or at least being able to interpret one from code. Below is a clean Python example that calculates whether the difference in conversion rates between a control and treatment group is statistically significant. This is the exact type of analysis you’d run after an A/B test at a company like PhonePe or Razorpay. Make sure you can read this code, explain every line, and modify it confidently.

import numpy as np
from scipy import stats

# --- A/B Test Results ---
# Control Group (Version A - Original checkout flow)
control_visitors = 10000
control_conversions = 850   # 8.5% conversion rate

# Treatment Group (Version B - New checkout flow with progress bar)
treatment_visitors = 10200
treatment_conversions = 960  # ~9.4% conversion rate

# --- Conversion Rates ---
p_control = control_conversions / control_visitors
p_treatment = treatment_conversions / treatment_visitors

print(f"Control Conversion Rate:   {p_control:.2%}")
print(f"Treatment Conversion Rate: {p_treatment:.2%}")
print(f"Relative Uplift:           {((p_treatment - p_control) / p_control):.2%}")

# --- Pooled proportion for Z-test ---
p_pooled = (control_conversions + treatment_conversions) / (control_visitors + treatment_visitors)

# --- Standard Error ---
se = np.sqrt(p_pooled * (1 - p_pooled) * (1/control_visitors + 1/treatment_visitors))

# --- Z-Score ---
z_score = (p_treatment - p_control) / se

# --- P-Value (two-tailed test) ---
p_value = 2 * (1 - stats.norm.cdf(abs(z_score)))

print(f"\nZ-Score:  {z_score:.4f}")
print(f"P-Value:  {p_value:.4f}")

# --- Decision at 95% confidence level (alpha = 0.05) ---
alpha = 0.05
if p_value < alpha:
    print(f"\n✅ Result: Statistically SIGNIFICANT at {(1-alpha)*100:.0f}% confidence.")
    print("   The new checkout flow performs significantly better. Consider shipping.")
else:
    print(f"\n❌ Result: NOT statistically significant at {(1-alpha)*100:.0f}% confidence.")
    print("   Insufficient evidence to conclude the new flow is better. Extend the test.")

# --- Minimum Sample Size Check (for future tests) ---
# Using formula: n = (Z_alpha + Z_beta)^2 * p*(1-p) / delta^2
# For 95% confidence, 80% power, 1% minimum detectable effect
from statsmodels.stats.power import NormalIndPower

effect_size = abs(p_treatment - p_control) / np.sqrt(p_pooled * (1 - p_pooled))
analysis = NormalIndPower()
required_n = analysis.solve_power(effect_size=effect_size,
                                   alpha=0.05,
                                   power=0.80,
                                   alternative='two-sided')

print(f"\n📊 Required sample size per group (80% power): {int(np.ceil(required_n))}")