Scenario-Based SRM

Question 1

Scenario: Assignment logs show 50/50, but only 35% of users in B reach checkout. Diagnosis?

Answer 1

Downstream logging failure, missing checkout events for bucket B, likely client version mismatch or ETL error.

Question 2

Scenario: SRM only for new users. Why?

Answer 2

Eligibility mismatch; new users filtered differently after assignment or logging delayed for new-user flows.

Question 3

Scenario: SRM appears only in EU region. Why?

Answer 3

Geo filters applied after assignment, regional logging outages, GDPR-driven opt-outs affecting one bucket more.

Question 4

Scenario: Treatment changes onboarding flow for new users. Can you use CUPED?

Answer 4

No — new users have no baseline; CUPED invalid.

Question 5

Scenario: Seasonality spike (Black Friday) occurs between pre-period and experiment. Use CUPED?

Answer 5

No — stationarity violated; baseline no longer predictive.

Question 6

Scenario: Pre-period metric exists, but is skewed for one region only.

Answer 6

Segment by region before applying CUPED.

Question 7

Scenario: pre-period correlation = 0.10.

Answer 7

CUPED useless — correlation too low; ignore CUPED.