Microsoft’s experimentation research describes SRM as a warning that should be diagnosed before interpreting effects. This guide adds an original, synthetic two-group exercise. It is separate from our A/B-testing conversion example: that guide compares outcomes; this one checks group counts.

A checkout experiment with an unexpected split

Suppose a retailer serving US and UK customers randomly assigns eligible users equally to two checkout experiences. At the planned analysis checkpoint, the assignment extract contains 550 distinct users in A and 450 in B. Each user appears once and belongs to one group. The planned split is 50/50, so the expected counts are 500 and 500.

For this exercise, the team chose an investigation threshold of 0.01 before inspecting the data. This is an illustrative choice, not a universal industry threshold. Production monitoring needs a defined policy for repeated checks and multiple experiments.

Run the Python check

The following standard-library script calculates Pearson’s chi-square statistic and its approximate upper-tail probability for exactly two groups. It assumes independent assignment units and fixed allocation probabilities. Use integer counts of distinct units, not pageviews or conversions.

from math import erfc, sqrt, isclose

def srm(a, b, share_a=0.5):
    if not (0 < share_a < 1) or min(a, b) < 0 or a+b == 0:
        raise ValueError("Check counts and planned allocation")
    n = a+b
    expected = (n*share_a, n-n*share_a)
    if min(expected) < 5:
        raise ValueError("Use a suitable exact test for small counts")
    chi2 = sum((o-e)**2/e for o,e in zip((a,b), expected))
    # Chi-square survival probability with exactly 1 degree of freedom.
    p = erfc(sqrt(chi2/2))
    return chi2, p

stat, p = srm(550, 450)
assert isclose(stat, 10)
assert 0.0015 < p < 0.0016
assert srm(500, 500) == (0.0, 1.0)
assert srm(800, 200, 0.8) == (0.0, 1.0)
print(f"chi2={stat:.2f}; p={p:.6f}")
print("Investigate:", p < 0.01)

Expected output: chi2=10.00; p=0.001565; Investigate: True. The statistic is (550−500)²/500 + (450−500)²/500 = 10. Under the stated null and assumptions, the result is unlikely enough to cross the exercise’s 0.01 threshold.

The SciPy chi-square reference explains the goodness-of-fit test and count-size limitations. Our two categories give one degree of freedom because the allocation is specified rather than estimated. The erfc shortcut here applies to that one-degree case only; do not reuse it for three or more variants. For small observed or expected counts, use an appropriate exact procedure instead of relying on this approximation.

Why 80/20 can be correct

The third assertion uses 800 users in A, 200 in B and a planned A share of 0.8. The counts match that design, so the statistic is zero. Calling every unequal split a defect would flag this valid allocation. Read the experiment configuration first, including any changes during rollout. If allocation changed midway, do not assume one fixed ratio describes every assignment.

Investigate where users disappeared

1. Start with assignment: compare counts from the randomization log before joining outcomes. Check duplicate IDs and users assigned to both variants.
2. Trace exposure: compare assigned users with recorded exposures. A missing event after a redirect might affect one variant more than the other.
3. Review exclusions: inspect eligibility rules, joins and missing-value filters. An inner join to purchases drops nonbuyers and creates a different population.
4. Locate the change: examine the timing of deployments and logging incidents. Treat country and device splits as diagnostic clues, accounting for the extra tests.
5. Resolve before concluding: document the defect, affected interval and recovery plan. A clean rerun may be needed if valid outcomes cannot be reconstructed.

For a cross-market rollout, confirm that regional eligibility and exposure rules follow the design. A US-versus-UK difference does not itself prove a country effect or identify the root cause. Preserve the original counts and explain which stage introduced the discrepancy.

Do not repair the chart by deleting users

Randomly deleting 100 users from A would make the displayed split equal but would not explain the mismatch. Likewise, reweighting without understanding the selection process does not establish a valid treatment comparison. First identify whether the issue is assignment, logging, filtering or the population definition.

A non-alert is not a certificate that the experiment is valid. Equal losses in both variants may preserve the ratio. Also check outcome completeness, stable identifiers, correct treatment delivery and the observation window. Use the retention eligibility exercise for a separate example of incomplete follow-up.

How to explain this in an interview

“I would verify the planned allocation and count distinct randomization units before joining outcomes. For this 50/50 example, 550 versus 450 triggers our predefined check. I would investigate the data pipeline before presenting lift. The SRM p-value does not tell us the probability that B is better.”

Continue with Python interview practice, merge validation and business case studies. Review preparation plans and current availability; all mock interviews and feedback are in English.