Hypothesis Testing & p-values

Question 1

What is the basic goal of hypothesis testing?

Answer 1

To assess whether observed data are compatible with a specified null hypothesis about a population or model parameter.

Question 2

What is a null hypothesis H₀?

Answer 2

A default assumption or claim about a parameter or distribution, such as ‘no effect’ or ‘no difference’ between groups.

Question 3

What is an alternative hypothesis H₁?

Answer 3

A competing claim that would be supported if the evidence is strong enough against the null, such as ‘there is a difference’.

Question 4

What is a test statistic in hypothesis testing?

Answer 4

A function of the data that measures the degree of discrepancy between the observed data and what would be expected under H₀.

Question 5

What is a p-value?

Answer 5

The probability, under the null hypothesis, of observing a test statistic at least as extreme as the one computed from the data.

Question 6

Does a small p-value prove that the null hypothesis is false?

Answer 6

No; it indicates that the observed data would be unusual under H₀, but does not provide a direct probability that H₀ is true or false.

Question 7

What is a significance level α (alpha)?

Answer 7

A chosen threshold such as 0.05, representing the maximum tolerable probability of rejecting H₀ when it is actually true (Type I error).

Question 8

What is a Type I error?

Answer 8

Rejecting a true null hypothesis (a false positive).

Question 9

What is a Type II error?

Answer 9

Failing to reject a false null hypothesis (a false negative).

Question 10

What is statistical power?

Answer 10

The probability of correctly rejecting H₀ when it is false; i.e., 1 − (Type II error rate).

Question 11

Why is power important in experimental design?

Answer 11

Low power means that even real effects are unlikely to be detected, leading to wasted experiments and misleading ‘no effect’ conclusions.

Question 12

What factors increase the power of a test?

Answer 12

Larger sample size, larger true effect size, lower variance, and higher significance level (all else equal).

Question 13

Why is ‘p<0.05’ not a magic threshold?

Answer 13

It is a convention; real decisions should consider effect size, uncertainty, costs of errors, and context, not just a binary cutoff.

Question 14

What is multiple testing or multiple comparisons?

Answer 14

Running many hypothesis tests, which increases the chance of getting at least one small p-value by random chance even when all nulls are true.

Question 15

Why is multiple testing a concern in ML and analytics?

Answer 15

Trying many models, features, or metrics and only reporting the best results can lead to overoptimistic conclusions and p-hacking.

Question 16

What is an A/B test at a high level?

Answer 16

A controlled experiment comparing outcomes between a control group (A) and a treatment group (B) to assess the effect of a change.

Question 17

What is randomization in A/B testing?

Answer 17

Assigning units to groups at random so that, on average, groups are comparable and confounders are balanced.

Question 18

Why is randomization critical for causal interpretation?

Answer 18

It breaks systematic links between treatment assignment and other variables, making differences in outcomes attributable to the treatment with fewer assumptions.

Question 19

What are typical units of randomization in online A/B tests?

Answer 19

Users, sessions, or requests, depending on the product and outcome being measured.

Question 20

Why is ‘stable unit treatment value’ (roughly) important in experiments?

Answer 20

It assumes one unit’s outcome is not affected by another unit’s treatment, simplifying interpretation; interference can complicate A/B test analysis.

Question 21

What is a lift in A/B testing?

Answer 21

The relative change in a metric between treatment and control, often expressed as a percentage increase or decrease.

Question 22

What is the difference between statistical significance and practical significance?

Answer 22

Statistical significance is about whether an effect is unlikely under H₀; practical significance is whether the effect size is large enough to matter in practice.

Question 23

Why is it dangerous to stop an experiment as soon as p<0.05 without planning?

Answer 23

Repeated peeking at results inflates Type I error; you effectively perform multiple tests without correction.

Question 24

What is a pre-analysis plan or fixed-horizon test?

Answer 24

A plan that specifies sample size, metrics, and decision rules in advance and analyzes data only after reaching the planned sample size.

Question 25

What is sequential testing or online testing?

Answer 25

Approaches that allow monitoring results over time with statistical corrections to maintain valid error rates when checking repeatedly.

Question 26

Why is sample size planning important before running an A/B test?

Answer 26

To ensure that the experiment has enough power to detect the minimum effect size of interest without running excessively long or short.

Question 27

What inputs are typically needed for rough sample size calculations?

Answer 27

Baseline metric level, minimum detectable effect size, desired power, and significance level.

Question 28

What role does variance play in experiment design?

Answer 28

Higher variance in the metric requires larger sample size to detect the same effect with the same power.

Question 29

Why can observational differences between groups be misleading without randomization?

Answer 29

Groups may differ in many ways besides the treatment, so differences in outcomes may be due to confounders rather than the treatment itself.

Question 30

What is a confounder?

Answer 30

A variable that influences both the treatment and the outcome, potentially creating a spurious association.

Question 31

What is a simple way to reduce confounding in observational analysis?

Answer 31

Control for known confounders via stratification, matching, or regression, while recognizing that unknown confounding may remain.

Question 32

Why are experiments still preferred when feasible, even with advanced observational methods?

Answer 32

Experiments provide cleaner identification of causal effects under simpler, more transparent assumptions.

Question 33

In ML model evaluation, what is an informal 'hypothesis test'?

Answer 33

Assessing whether one model’s performance metric is substantially better than another’s beyond what could be expected from random variation in the test set.

Question 34

Why is it useful to think of model comparisons as noisy estimates rather than exact ranks?

Answer 34

Performance metrics on finite data have sampling variability; small differences may be indistinguishable from noise.

Question 35

What is a good practice when comparing models using metrics?

Answer 35

Report point estimates with confidence intervals or uncertainty estimates, and consider business impact, not just statistical significance.

Question 36

What is p-hacking?

Answer 36

Manipulating analysis choices or repeatedly testing until a desired p-value is obtained, inflating false discovery rates.

Question 37

How can teams reduce p-hacking risk?

Answer 37

Pre-registering analysis plans, limiting post-hoc choices, correcting for multiple comparisons, and emphasizing effect sizes and uncertainty.

Question 38

In one sentence, what should an ML engineer remember about hypothesis tests and p-values?

Answer 38

They are tools for quantifying evidence under a model, not definitive answers; combine them with effect sizes, uncertainty, and domain context when making decisions.