Experimental Design & A/B Testing

Question 1

What is the main goal of an A/B test?

Answer 1

To compare outcomes between a control (A) and a treatment (B) group in a randomized experiment to estimate the causal effect of a change.

Question 2

What is randomization in A/B testing?

Answer 2

Assigning units (such as users or sessions) to treatment or control at random so that, on average, the groups are comparable on all other factors.

Question 3

Why is randomization critical for causal interpretation?

Answer 3

It breaks systematic links between treatment assignment and confounders, so differences in outcomes can be attributed to the treatment under reasonable assumptions.

Question 4

What are common units of randomization in online experiments?

Answer 4

Users, sessions, or requests, chosen so that interference between units is minimized and exposure is clearly defined.

Question 5

What is a treatment effect at a high level?

Answer 5

The difference in expected outcomes between the treatment and control conditions for the same population.

Question 6

What is uplift or lift in A/B tests?

Answer 6

The relative change in a metric between treatment and control, often expressed as a percentage increase or decrease.

Question 7

What is a primary metric in experimental design?

Answer 7

The main outcome of interest that the experiment is aimed at improving, such as conversion rate or retention.

Question 8

What are guardrail metrics?

Answer 8

Metrics monitored to ensure that the treatment does not harm important aspects of the product, such as latency or error rates.

Question 9

What is a null hypothesis H₀ in the context of an A/B test?

Answer 9

The assumption that there is no difference in the metric between treatment and control (effect size zero).

Question 10

What is an alternative hypothesis H₁ in an A/B test?

Answer 10

The claim that there is a nonzero difference (positive or negative) between treatment and control.

Question 11

What is a significance level α in experiment design?

Answer 11

A pre-specified maximum probability of rejecting a true null hypothesis (Type I error), often chosen as 0.05.

Question 12

What is statistical power in an experiment?

Answer 12

The probability of correctly rejecting the null hypothesis when the treatment truly has the minimum effect size of interest.

Question 13

Why is power important when planning experiments?

Answer 13

Low power means a high chance of missing real effects, leading to wasted experimentation and misleading ‘no effect’ conclusions.

Question 14

What factors influence the required sample size for a given power?

Answer 14

Baseline metric level, minimum detectable effect size, variance, desired power, and chosen significance level.

Question 15

Why does smaller minimum detectable effect size require larger sample size?

Answer 15

Detecting subtler differences requires more data to distinguish them from random noise in metrics.

Question 16

What is a two-sample test for proportions in an A/B test?

Answer 16

A statistical test that compares conversion or success rates between treatment and control using binomial or Normal approximations.

Question 17

Why are confidence intervals often more informative than just p-values in A/B tests?

Answer 17

They show a range of plausible effect sizes and convey both magnitude and uncertainty, not just significance yes/no.

Question 18

What is a fixed-horizon (classical) A/B test?

Answer 18

An experiment where the sample size and analysis time are set in advance and results are evaluated only after data collection is complete.

Question 19

Why does repeatedly checking p-values before the planned sample size inflate Type I error?

Answer 19

Each additional look at the data effectively adds another hypothesis test, increasing the chance of false positives unless corrected.

Question 20

What is sequential or online testing?

Answer 20

Approaches that allow monitoring results over time with statistical corrections to maintain error control while peeking at data.

Question 21

What is a simple good practice for stopping rules if you use fixed-horizon tests?

Answer 21

Commit to a sample size and analysis plan in advance and avoid acting on interim p-values unless using a sequential method.

Question 22

What is a one-sided vs two-sided test in A/B experiments?

Answer 22

A one-sided test evaluates improvement in a specific direction; a two-sided test looks for any difference, positive or negative.

Question 23

Why are two-sided tests often preferable in product experiments?

Answer 23

They guard against harmful changes by detecting significant decreases as well as increases in the metric.

Question 24

What is a pre-analysis plan?

Answer 24

A document or specification that defines hypotheses, metrics, sample size, and analysis methods before running the experiment.

Question 25

How does a pre-analysis plan help avoid p-hacking?

Answer 25

It commits the team to specific analyses, reducing temptation to search for significance through many unplanned tests.

Question 26

What is multiple testing in the context of experiments?

Answer 26

Running many experiments or testing many metrics/hypotheses, which increases the chance of at least one false positive.

Question 27

What are simple strategies to mitigate multiple testing issues?

Answer 27

Limiting the number of primary metrics, adjusting p-values (e.g., Bonferroni), or emphasizing effect sizes and replication over single p-values.

Question 28

Why is it important to distinguish statistical significance from practical significance?

Answer 28

A tiny but statistically significant effect may be irrelevant to the business, while a practically important effect may be nonsignificant if the study is underpowered.

Question 29

What is sample ratio mismatch (SRM) in A/B tests?

Answer 29

A situation where the observed fraction of users in each group deviates significantly from the planned allocation ratio, often indicating randomization or logging issues.

Question 30

Why is detecting SRM important?

Answer 30

Because it can signal bugs in assignment, tracking, or eligibility logic that invalidate experiment results.

Question 31

What is interference between units in experiments?

Answer 31

When the treatment of one unit (e.g., user) affects outcomes of another, violating independence assumptions.

Question 32

How can interference complicate A/B test interpretation?

Answer 32

It breaks the simple comparison of group averages as causal effects, since spillovers can bias differences in either direction.

Question 33

What is a crossover or contamination in A/B tests?

Answer 33

When units switch between treatment and control conditions or are exposed to both, blurring group distinctions.

Question 34

Why should exposure and assignment logs be carefully instrumented?

Answer 34

Accurate logs are essential for understanding who saw what when, diagnosing SRM, and defining correct analysis populations.

Question 35

What is an intention-to-treat (ITT) analysis?

Answer 35

An analysis that compares groups based on initial random assignment regardless of actual treatment uptake, preserving randomization.

Question 36

Why is ITT often preferred over per-protocol analysis?

Answer 36

It avoids selection bias introduced by post-assignment behavior and maintains the benefits of randomization.

Question 37

What is a holdout group in experimentation?

Answer 37

A group of users withheld from any new changes, used as a long-term baseline for monitoring overall system trends.

Question 38

Why can long-lived holdouts be valuable?

Answer 38

They help distinguish experiment-driven changes from secular trends or seasonal effects in metrics.

Question 39

In the context of ML, what is offline evaluation vs online A/B testing?

Answer 39

Offline evaluation uses historical data and metrics on test sets; online A/B testing measures real-world impact on live traffic and business outcomes.

Question 40

Why can an ML model that looks better offline fail in an A/B test?

Answer 40

Offline metrics may not capture feedback effects, selection changes, latency, or user behavior shifts that appear in production.

Question 41

In one sentence, what is the key mental model for experimental design and A/B testing?

Answer 41

Plan experiments with clear hypotheses, metrics, and sample sizes, randomize carefully, monitor for pathologies like SRM, and interpret results by combining statistical evidence with practical impact.