Advanced SRM

Question 2

Does a small imbalance (e.g., 51% vs 49%) automatically mean SRM?

Answer 2

No — small imbalances can occur by chance. Only a chi-square test confirms SRM.

Question 3

Should chi-square for SRM be run on user-level assignments or on events?

Answer 3

User-level assignments. Event-level counts inflate sample size and give false positives.

Question 4

SRM appears only in conversion events, not assignment events. What does this mean?

Answer 4

Assignment is correct; downstream logging, joins, or ETL ingestion is failing for certain buckets.

Question 5

Your experiment shows perfect 50/50 assignment but checkout events show 60/40. Why?

Answer 5

Event logs failing, client-side tracking broken, or differences in event firing by bucket; pipeline joins inconsistent.

Question 6

SRM appears only on mobile traffic, not desktop. Likely cause?

Answer 6

Mobile app version mismatch, client logging missing, or mobile-specific caching/stickiness.

Question 7

SRM reversed mid-experiment (A bigger first 3 days, B bigger last 3 days). What does it suggest?

Answer 7

Experiment rollout issues, partial ramping, deployment/configuration change, or logging regression introduced midweek.

Question 8

Assignment is correct but session counts are wrong by bucket. Why?

Answer 8

Sticky sessions or caching routes certain users disproportionately to one bucket.

Question 9

If filtering is applied after assignment (e.g., eligible users only), what happens?

Answer 9

Filtering breaks randomization → SRM because buckets have different eligibility distributions.

Question 10

Bots hit one bucket much more due to caching. How does this affect SRM?

Answer 10

Artificial SRM — bot traffic distorts assignment ratios even if human assignment is correct.

Question 11

Is the presence of SRM always a showstopper?

Answer 11

Yes — you cannot interpret experiment results until SRM cause is fixed.