A fictional checkout experiment

Each group contains 10,000 randomly assigned users observed for the same period. Control has 500 buyers and 20 refunds; treatment has 580 buyers and 35 refunds. Assume each buyer made one purchase and each refund relates to one buyer. Conversion improves from 5% to 5.8%, a difference of 0.8 percentage points or 16% relative lift.

Choose the denominator deliberately

Refunds per buyer is operationally meaningful, but purchase is itself affected by the treatment. Comparing only buyers conditions on a post-treatment outcome. Refunds per assigned user keeps the original randomised population. Report both with clear labels and avoid interpreting them as interchangeable causal quantities.

Define a decision rule before launch

A hypothetical rule could require a practically useful conversion improvement while limiting deterioration in refund incidence and support burden. The actual margins should come from business costs and user experience, not from the observed result. Include a minimum observation period so refunds and repeat use can mature.

With only these summary counts, we have not established whether the guardrail change is statistically conclusive. A non-significant result also would not prove safety. Choose an analysis suitable for the randomisation unit, metric distribution and pre-specified acceptable deterioration. Document uncertainty rather than converting every result into a pass or fail.

Before trusting the experiment

• Check assignment counts and whether exposure or logging differs by group.
• Verify that the same user stays in the same group.
• Compare equal follow-up windows and the same eligibility rules.
• Inspect missing events, duplicate purchases and concurrent experiments.
• Account for repeated peeking and multiple metrics in the analysis plan.

Interview recommendation

“Conversion improved descriptively, but refunds also increased. I would check follow-up completeness and the pre-agreed guardrail analysis before recommending rollout. If the evidence remains uncertain, we should gather enough information to judge the defined risk rather than ignoring the refund signal.”

Practice extension

Assume each retained purchase contributes $20 and a refunded purchase contributes zero before support costs. Control has 480 retained purchases and treatment 545, implying $9,600 versus $10,900 under those assumptions. Explain why this simplified arithmetic does not include future refunds, support workload or uncertainty, and specify what you would measure next.