Experiment Math Basics (Explicit)

Question 1

What is “difference” in an experiment?

Answer 1

The gap between treatment and control. Absolute: T − C. Relative: (T − C)/C.

Question 2

What is variance?

Answer 2

Measure of how spread out individual outcomes are. High variance = noisy experiment.

Question 3

What is Standard Error (SE)?

Answer 3

Uncertainty of the mean estimate. SE = sqrt(variance / sample size).

Question 4

What determines SE?

Answer 4

Higher variance → larger SE. Larger sample → smaller SE.

Question 5

What is a confidence interval (CI)?

Answer 5

A range that likely contains the true effect. CI = estimate ± (z × SE).

Question 6

What does CI width tell you?

Answer 6

Wide CI = noisy, underpowered. Narrow CI = precise, well-powered.

Question 7

What is a p-value?

Answer 7

Probability of seeing an effect this large if true effect were zero. Low p = strong evidence effect is real.

Question 8

What is the relationship between CI and p-value?

Answer 8

If CI crosses 0 → p > 0.05. If CI excludes 0 → p < 0.05.

Question 9

What is statistical power?

Answer 9

Ability to detect a true effect. Higher power from lower variance & larger sample.

Question 10

How does CUPED improve power?

Answer 10

Reduces variance → lowers SE → tightens CI → easier to detect effects.