Correlation, Dependence, and Causality

Question 1

What is correlation between two variables X and Y at a high level?

Answer 1

A standardized measure of the strength and direction of a linear relationship between X and Y.

Question 2

How is Pearson correlation coefficient defined in terms of covariance?

Answer 2

ρ(X,Y) = Cov(X,Y) / (σ_X σ_Y), where σ_X and σ_Y are the standard deviations of X and Y.

Question 3

What are the possible values of Pearson correlation?

Answer 3

Between −1 and 1 inclusive, where −1 is perfect negative linear, 0 is no linear correlation, and 1 is perfect positive linear.

Question 4

What does ρ(X,Y) = 0 imply about X and Y?

Answer 4

There is no linear correlation; they may still be dependent in a nonlinear way.

Question 5

Does high correlation imply that X causes Y?

Answer 5

No; correlation alone does not establish causality, only association.

Question 6

What is spurious correlation?

Answer 6

An observed correlation between variables that arises from coincidence or common causes rather than a direct relationship.

Question 7

What is a confounder in causal reasoning?

Answer 7

A variable that influences both the potential cause and the outcome, potentially creating a misleading association.

Question 8

How can confounding lead to incorrect causal conclusions?

Answer 8

If not controlled, changes in the confounder may be mistaken for causal effects of the variable of interest.

Question 9

Why is ‘correlation ≠ causation’ particularly important in ML contexts?

Answer 9

Models can capture patterns that are predictive but unstable or non-causal; deploying them can fail when underlying associations change.

Question 10

What is rank-based (Spearman) correlation at a high level?

Answer 10

A correlation measure based on ranks of the data, capturing monotonic relationships (not just linear) and being more robust to outliers.

Question 11

When might Spearman correlation be preferred over Pearson?

Answer 11

When relationships are monotonic but not linear, or when outliers and non-Normality make Pearson unreliable.

Question 12

What is partial correlation?

Answer 12

The correlation between two variables after removing the linear effects of one or more other variables.

Question 13

Why can partial correlation be more informative than raw correlation?

Answer 13

It helps isolate associations that are not explained by obvious third variables, though it still does not prove causality.

Question 14

What is dependence in probability theory?

Answer 14

Any situation where the joint distribution of variables does not factor into the product of their marginals; knowing one gives information about the other.

Question 15

Why is independence stronger than zero correlation?

Answer 15

Independence implies zero correlation for variables with finite variance, but zero correlation does not rule out nonlinear dependence.

Question 16

What is mutual information (MI) at a high level?

Answer 16

A nonnegative measure of how much knowing one variable reduces uncertainty about another, capturing general dependence, not just linear.

Question 17

Why is mutual information useful in feature selection?

Answer 17

It can detect nonlinear and non-monotonic relationships between features and targets.

Question 18

What is the danger of selecting features solely based on correlation with the label?

Answer 18

You may pick many redundant or spurious features and inflate false discoveries due to multiple testing.

Question 19

What is the difference between predictive and causal relationships?

Answer 19

Predictive relationships help forecast outcomes, while causal relationships describe how interventions on one variable would change another.

Question 20

Why can a non-causal feature still be useful in ML?

Answer 20

Even non-causal features can be strongly predictive if they are stable proxies or capture useful information about the outcome.

Question 21

When can using non-causal features be dangerous?

Answer 21

When underlying associations can change under new policies, environments, or behaviors, causing models to fail or behave unfairly.

Question 22

What is a causal effect at a high level?

Answer 22

The difference in an outcome that would occur under one intervention versus another, holding everything else constant in a conceptual experiment.

Question 23

What is the role of randomized experiments in discovering causal effects?

Answer 23

Random assignment breaks confounding, so differences in outcomes between groups can be interpreted as causal under reasonable assumptions.

Question 24

Why are randomized experiments not always feasible?

Answer 24

They can be expensive, unethical, or logistically impossible in some settings.

Question 25

What is an observational study?

Answer 25

An analysis of data where treatments or exposures are not randomized, so confounding must be addressed statistically.

Question 26

Why is causal inference from observational data more fragile than from experiments?

Answer 26

It relies on untestable assumptions about no unmeasured confounding and correct model specification.

Question 27

What is a directed acyclic graph (DAG) in causal modeling?

Answer 27

A graph where nodes are variables and directed edges represent assumed causal relationships, with no cycles.

Question 28

How can DAGs help reason about confounding and conditioning?

Answer 28

They make assumptions explicit, showing which variables to adjust for to block backdoor paths and avoid opening colliders.

Question 29

What is a backdoor path in a causal DAG?

Answer 29

A path from treatment to outcome that starts with an arrow into the treatment, typically representing confounding influences.

Question 30

What is the backdoor criterion (informally)?

Answer 30

A condition that identifies sets of variables to adjust for to block all backdoor paths and estimate causal effects.

Question 31

What is a collider in a DAG?

Answer 31

A node where two arrows meet head-to-head; conditioning on it can introduce spurious associations between its parents.

Question 32

Why is conditioning on colliders dangerous?

Answer 32

It can create dependence between variables that were previously independent, biasing causal estimates.

Question 33

What is Simpson’s paradox at a high level?

Answer 33

A phenomenon where an association reverses direction when data are aggregated versus stratified by a third variable.

Question 34

Why is Simpson’s paradox relevant to ML and analytics?

Answer 34

It illustrates how ignoring relevant stratification or confounders can lead to completely misleading conclusions from correlations.

Question 35

What is selection bias?

Answer 35

Bias introduced when the data we observe are not a random sample from the target population, often due to the selection mechanism.

Question 36

How can selection bias affect ML models?

Answer 36

Models can learn relationships that only hold in the selected sample and fail when deployed to a broader or different population.

Question 37

What is covariate shift in ML?

Answer 37

A change in the distribution of input features between training and deployment while the conditional distribution of output given input remains the same.

Question 38

How does covariate shift relate to correlation and dependence?

Answer 38

Correlations learned in the training distribution may change under covariate shift, affecting model performance.

Question 39

Why should ML engineers care about causal structure even if they only build predictive models?

Answer 39

Understanding causal structure helps anticipate which patterns are stable under interventions or policy changes and which may break.

Question 40

What is an example of a stable causal feature vs an unstable proxy?

Answer 40

Underlying health status may be a stable cause of hospital visits, while insurance plan might be a proxy that changes with policy or market shifts.

Question 41

What is the high-level idea of invariant risk minimization (IRM) and related causal ML methods?

Answer 41

To learn predictors whose relationships to the outcome remain stable across different environments, approximating causal predictors.

Question 42

Why is overfitting to spurious correlations a central risk in ML?

Answer 42

Models may latch onto patterns that are strong in historical data but vanish under new conditions, causing sharp drops in performance.

Question 43

In one sentence, what should you remember about correlation, dependence, and causality for ML?

Answer 43

Correlation and dependence tell you what predicts well in current data, but only causal reasoning and experiments can tell you which relationships will remain stable under change.