Core Probability & Random Variables

Question 1

What is a sample space in probability theory?

Answer 1

The set of all possible basic outcomes of a random experiment.

Question 2

What is an event in probability?

Answer 2

Any subset of the sample space, representing a collection of outcomes we care about.

Question 3

How is the probability of an event informally interpreted?

Answer 3

As a number between 0 and 1 representing how likely the event is to occur, with 0 impossible and 1 certain.

Question 4

What are the three basic properties any probability measure must satisfy?

Answer 4

Non-negativity, P(sample space)=1, and additivity for mutually exclusive events (P(A ∪ B)=P(A)+P(B) when A and B cannot both occur).

Question 5

What is the complement of an event A?

Answer 5

The event consisting of all outcomes in the sample space that are not in A.

Question 6

How is the probability of the complement of A related to P(A)?

Answer 6

P(Aᶜ) = 1 − P(A).

Question 7

What does it mean for two events A and B to be mutually exclusive (disjoint)?

Answer 7

They cannot happen at the same time, so their intersection is empty and P(A ∩ B)=0.

Question 8

What is the general addition rule for probabilities of two events A and B?

Answer 8

P(A ∪ B) = P(A) + P(B) − P(A ∩ B).

Question 9

What is conditional probability P(A|B)?

Answer 9

The probability that event A occurs given that event B has occurred, defined as P(A ∩ B) / P(B) when P(B) > 0.

Question 10

How can you interpret P(A|B) intuitively?

Answer 10

It is the probability of A when we restrict our attention to the subset of outcomes where B has happened.

Question 11

What is the multiplication rule relating joint and conditional probability?

Answer 11

P(A ∩ B) = P(A|B) · P(B) = P(B|A) · P(A).

Question 12

What does it mean for two events A and B to be independent?

Answer 12

Knowing that one occurs gives no information about the other, so P(A ∩ B) = P(A)P(B) and P(A|B) = P(A).

Question 13

Can events be mutually exclusive and independent at the same time (with nonzero probabilities)?

Answer 13

No; if they are mutually exclusive and both have nonzero probability, then P(A ∩ B)=0≠P(A)P(B).

Question 14

What is Bayes’ theorem for two events A and B?

Answer 14

P(A|B) = P(B|A)P(A) / P(B), assuming P(B) > 0.

Question 15

Why is Bayes’ theorem important in ML contexts?

Answer 15

It provides a way to invert conditional probabilities and update beliefs about hypotheses given observed evidence.

Question 16

What is the law of total probability?

Answer 16

If {B₁,…,Bₙ} is a partition of the sample space, then P(A) = Σ P(A|Bᵢ)P(Bᵢ).

Question 17

What is a random variable?

Answer 17

A function that maps outcomes in the sample space to numeric values, allowing us to analyze numerical aspects of randomness.

Question 18

What is the difference between a discrete and a continuous random variable?

Answer 18

A discrete random variable takes values in a countable set; a continuous random variable takes values in an interval or continuum of real numbers.

Question 19

What is a probability mass function (pmf)?

Answer 19

A function p(x) that gives P(X = x) for a discrete random variable X.

Question 20

What two key properties must a pmf satisfy?

Answer 20

p(x) ≥ 0 for all x, and the sum over all possible x of p(x) equals 1.

Question 21

What is a probability density function (pdf)?

Answer 21

A nonnegative function f(x) such that the probability that a continuous random variable X lies in an interval is given by the integral of f over that interval.

Question 22

Why does it not make sense to ask for P(X = x) for a continuous random variable with a pdf?

Answer 22

For a continuous variable, the probability at any specific point is typically 0; probabilities are associated with intervals, not points.

Question 23

What is a cumulative distribution function (cdf)?

Answer 23

A function F(x) = P(X ≤ x) that gives the probability that the random variable X is less than or equal to x.

Question 24

How is the cdf of a discrete random variable related to its pmf?

Answer 24

F(x) is the sum of p(t) over all t ≤ x.

Question 25

How is the cdf of a continuous random variable related to its pdf?

Answer 25

F(x) is the integral of f(t) from −∞ up to x.

Question 26

What does it mean for two random variables X and Y to be identically distributed?

Answer 26

They share the same probability distribution (same pmf or pdf), even if they are not connected or observed together.

Question 27

What does it mean for random variables X₁,…,Xₙ to be independent?

Answer 27

For any selection of values, the joint probability factorizes into the product of the individual probabilities, e.g., P(X₁=x₁,…,Xₙ=xₙ)=∏ P(Xᵢ=xᵢ).

Question 28

What is a joint distribution of two random variables X and Y?

Answer 28

A function (pmf or pdf) that gives probabilities (or densities) for all pairs (x,y) together, not just individually.

Question 29

What is a marginal distribution?

Answer 29

The distribution of one variable obtained from a joint distribution by summing or integrating over the other variable(s).

Question 30

How do you obtain the marginal pmf of X from a joint pmf of X and Y?

Answer 30

Sum the joint pmf over all possible values of Y: p_X(x) = Σ_y p_{X,Y}(x,y).

Question 31

How do you obtain the marginal pdf of X from a joint pdf of X and Y?

Answer 31

Integrate the joint pdf over all values of Y: f_X(x) = ∫ f_{X,Y}(x,y) dy.

Question 32

What is the conditional distribution of Y given X=x?

Answer 32

A distribution with pmf or pdf proportional to the joint distribution at X=x, normalized so probabilities sum or integrate to 1.

Question 33

How is conditional pmf p(Y=y | X=x) related to the joint and marginal pmfs?

Answer 33

p(Y=y | X=x) = p_{X,Y}(x,y) / p_X(x), assuming p_X(x) > 0.

Question 34

How is the concept of independence expressed in terms of joint and marginal distributions?

Answer 34

X and Y are independent if and only if their joint distribution factorizes: p_{X,Y}(x,y) = p_X(x)p_Y(y) or f_{X,Y}(x,y)=f_X(x)f_Y(y).

Question 35

What is the expectation (mean) of a random variable at a high level?

Answer 35

The long-run average value of the variable if we could repeat the random experiment many times.

Question 36

How is the expectation of a discrete random variable X with pmf p(x) defined?

Answer 36

E[X] = Σ x · p(x), summing over all possible values x.

Question 37

How is the expectation of a continuous random variable X with pdf f(x) defined?

Answer 37

E[X] = ∫ x · f(x) dx, integrating over the support of X.

Question 38

What does linearity of expectation mean?

Answer 38

For any random variables X and Y and constants a, b, E[aX + bY] = aE[X] + bE[Y], regardless of whether X and Y are independent.

Question 39

What is the variance of a random variable at a high level?

Answer 39

A measure of how much the variable’s values spread around its mean.

Question 40

How is variance Var(X) defined in terms of expectation?

Answer 40

Var(X) = E[(X − E[X])²].

Question 41

What is the standard deviation of a random variable?

Answer 41

The square root of the variance, providing spread in the same units as the original variable.

Question 42

What is covariance between two random variables X and Y (intuitively)?

Answer 42

A measure of how X and Y vary together, positive if they tend to go up and down together, negative if they move in opposite directions.

Question 43

Why is correlation often used instead of raw covariance?

Answer 43

Correlation normalizes covariance to lie between −1 and 1, making it easier to interpret strength and direction of linear relationships.

Question 44

Why does zero covariance (or correlation) not necessarily imply independence?

Answer 44

Because two variables can have a nonlinear relationship that makes covariance zero, yet still be statistically dependent in a more complex way.