Advanced CUPED

Question 1

What happens if CUPED baseline (X) has missing values for some users?

Answer 1

CUPED invalid for those users; either drop them or use alternative adjustment.

Question 2

Why must users exist in both pre-period and experiment period for CUPED?

Answer 2

CUPED compares each user’s baseline to their experiment metric; new users have no X → CUPED unusable.

Question 3

What happens if correlation between X and Y differs across segments (e.g., device, geography)?

Answer 3

Apply CUPED separately within segments; mixed correlations cause suboptimal variance reduction.

Question 4

Does CUPED change the estimated treatment effect?

Answer 4

No — it only reduces variance (narrower CIs); the mean effect remains unchanged.

Question 5

If correlation is high in pre-period but unstable over time (seasonality), should CUPED be used?

Answer 5

No — metric non-stationarity breaks CUPED assumptions.

Question 6

What is baseline contamination?

Answer 6

Treatment affecting the pre-period metric, making X invalid for variance reduction.

Question 7

Pre-period logs were partially missing. Should you apply CUPED?

Answer 7

No — incomplete baseline logging introduces bias into adjusted metric.

Question 8

Why can CUPED dramatically reduce required sample size?

Answer 8

Lower metric variance → more power → smaller minimum detectable effect → shorter experiment.

Question 9

What is the main assumption behind CUPED?

Answer 9

Pre-period metric X predicts part of the variation in experiment metric Y, and treatment does not affect X.

Question 10

If correlation(X,Y)=0.75, what variance reduction can CUPED offer?

Answer 10

Roughly ~50–60% variance reduction (approx. correlation²).