SRM

Question 1

SRM

Answer 1

What does “Sample Ratio Mismatch” mean? || Actual bucket sizes differ significantly from the intended experiment split (e.g., expected 50/50, observed 60/40).

Question 2

Why is SRM dangerous in A/B tests?

Answer 2

It invalidates the entire experiment because assignment or logging is broken; results cannot be trusted.

Question 3

How do you detect SRM?

Answer 3

Run a chi-square goodness-of-fit test comparing observed vs expected bucket counts.

Question 4

If SRM exists, should you continue analyzing results?

Answer 4

No — stop immediately. Results are invalid.

Question 5

Top cause of SRM #1

Answer 5

Randomization bugs (bad hashing, split logic errors).

Question 6

Top cause of SRM #2

Answer 6

Missing or incomplete logs (assignment logs fail, dropped events, ETL skips).

Question 7

Top cause of SRM #3

Answer 7

Eligibility filters applied after assignment (geo/device/user type filtering).

Question 8

Top cause of SRM #4

Answer 8

Platform issues (caching, sticky sessions, version mismatches).

Question 9

What is a realistic tiny imbalance that is not SRM?

Answer 9

Small natural variation (e.g. 50.3% vs 49.7%) — chi-square will confirm it’s random.

Question 10

In a 70/30 test with observed 80/20, is this SRM?

Answer 10

Yes — extreme imbalance, fails chi-square, experiment invalid.