CUPED

Question 1

What is CUPED?

Answer 1

A variance-reduction technique that adjusts metrics using pre-experiment baseline values.

Question 2

Why does CUPED make A/B tests faster?

Answer 2

Because it reduces variance, increasing statistical power → fewer samples needed → shorter experiment.

Question 3

What does the CUPED adjustment formula look like (intuitively)?

Answer 3

Adjusted metric = Y − θX, where Y is experiment metric, X is pre-period metric.

Question 4

What is θ in CUPED?

Answer 4

θ = covariance(Y, X) / variance(X); learned via linear regression.

Question 5

What must be true for CUPED to work well?

Answer 5

High correlation between pre-period (X) and experiment metric (Y).

Question 6

When should you NOT use CUPED?

Answer 6

When the treatment affects the baseline (contamination), new users with no history, seasonal shifts, or broken pre-period logs.

Question 7

What is “baseline contamination” in CUPED?

Answer 7

When the treatment leaks into or affects the pre-period metric, making CUPED invalid.

Question 8

What is the main benefit of CUPED?

Answer 8

Narrower confidence intervals → increased sensitivity → reduced minimum detectable effect.

Question 9

If correlation between baseline and metric is 0.75, should CUPED be used?

Answer 9

Yes — strong correlation → large variance reduction.

Question 10

If correlation is low (e.g. 0.05), should CUPED be used?

Answer 10

No — low correlation gives little benefit and may add noise.