Bayesian Basics for ML

Question 1

What is the core idea of Bayesian inference?

Answer 1

To treat unknown parameters as random variables and update a prior belief about them using observed data to obtain a posterior belief.

Question 2

In Bayesian terms, what is a prior distribution?

Answer 2

A probability distribution that represents our beliefs about a parameter before seeing the current data.

Question 3

What is a likelihood in Bayesian inference?

Answer 3

The probability of the observed data as a function of the parameter, reflecting how plausible the data are under different parameter values.

Question 4

What is a posterior distribution?

Answer 4

The updated distribution of the parameter after combining the prior and the likelihood using Bayes’ rule.

Question 5

What is Bayes’ rule in the parameter-data form?

Answer 5

Posterior ∝ Likelihood × Prior, or p(θ|data) ∝ p(data|θ)p(θ).

Question 6

What does the normalization constant in Bayes’ rule ensure?

Answer 6

That the posterior distribution integrates (or sums) to 1 over all possible parameter values.

Question 7

What is a point estimate in the Bayesian framework?

Answer 7

A single summary of the posterior such as the posterior mean, median, or maximum a posteriori (MAP) estimate.

Question 8

What is the MAP (maximum a posteriori) estimate?

Answer 8

The parameter value that maximizes the posterior distribution p(θ|data).

Question 9

How is MAP estimation related to regularized maximum likelihood?

Answer 9

MAP is equivalent to maximizing log-likelihood plus log-prior, which often looks like a regularized loss function.

Question 10

What type of prior corresponds to L2 regularization in linear models?

Answer 10

A Normal (Gaussian) prior on coefficients, leading to ridge-like penalties.

Question 11

What type of prior corresponds to L1 regularization in linear models?

Answer 11

A Laplace (double-exponential) prior on coefficients, leading to lasso-like penalties.

Question 12

Why is it useful to view regularization as a Bayesian prior?

Answer 12

It provides an interpretation of regularization as encoding prior beliefs about parameter magnitude and supports probabilistic reasoning.

Question 13

What is a conjugate prior?

Answer 13

A prior distribution chosen so that the posterior is in the same family as the prior when combined with a given likelihood.

Question 14

Why are conjugate priors convenient?

Answer 14

They allow closed-form posterior updates, simplifying analytic calculations and reducing computational cost.

Question 15

What is a conjugate prior for a Bernoulli or Binomial likelihood?

Answer 15

The Beta distribution is conjugate to Bernoulli and Binomial likelihoods for a probability parameter.

Question 16

What is a conjugate prior for a Poisson likelihood?

Answer 16

The Gamma distribution is conjugate for the rate parameter of a Poisson likelihood.

Question 17

What is a conjugate prior for a Normal likelihood with known variance and unknown mean?

Answer 17

A Normal prior on the mean is conjugate, yielding a Normal posterior for the mean.

Question 18

What is a Bayesian credible interval?

Answer 18

An interval [a,b] such that the posterior probability that the parameter lies in [a,b] equals a chosen level (e.g., 95%).

Question 19

How does a credible interval differ conceptually from a frequentist confidence interval?

Answer 19

A credible interval directly expresses probability about the parameter given the data; a confidence interval concerns the long-run coverage of the procedure under repeated sampling.

Question 20

What is the posterior predictive distribution?

Answer 20

The distribution of a future observation, obtained by averaging the likelihood of new data over the posterior distribution of the parameters.

Question 21

Why is the posterior predictive distribution useful in ML?

Answer 21

It captures both parameter uncertainty and data noise, providing more realistic uncertainty about future predictions.

Question 22

What is hierarchical (multilevel) modeling in the Bayesian context?

Answer 22

Modeling parameters themselves as drawn from higher-level distributions, allowing sharing of information across groups or entities.

Question 23

Why are hierarchical models powerful?

Answer 23

They enable partial pooling across groups, improving estimates for small-sample groups and capturing structure in multi-level data.

Question 24

What is partial pooling?

Answer 24

An approach where group-specific estimates are shrunk towards a global mean based on data and prior, balancing between no pooling and complete pooling.

Question 25

What is a posterior mean estimator for a probability parameter under a Beta–Binomial model?

Answer 25

Often a weighted average of the prior mean and observed proportion, with weights depending on prior strength and sample size.

Question 26

Why can Bayesian updating be seen as a form of 'smart smoothing' of empirical estimates?

Answer 26

Priors temper extreme values from small samples, pulling estimates towards plausible ranges until more data reduces prior influence.

Question 27

What role do priors play when we have lots of data?

Answer 27

Under regular conditions, the likelihood dominates and the influence of reasonable priors diminishes as sample size grows.

Question 28

Why can prior choice matter a lot in small-sample settings?

Answer 28

With limited data, the prior contributes substantially to the posterior, affecting estimates and uncertainty.

Question 29

What is a noninformative or weakly informative prior?

Answer 29

A prior chosen to have minimal influence or to rule out only extreme or implausible parameter values while remaining relatively diffuse.

Question 30

Why are truly 'noninformative' priors often elusive?

Answer 30

Priors that appear flat in one parameterization can be highly informative under reparameterization, and boundaries/constraints complicate neutrality.

Question 31

What is MCMC (Markov chain Monte Carlo) in Bayesian computation?

Answer 31

A class of algorithms that approximate posterior distributions by drawing samples via a Markov chain whose stationary distribution is the posterior.

Question 32

Why is MCMC needed for many Bayesian models?

Answer 32

Because closed-form posteriors are unavailable and direct integration is intractable in high dimensions.

Question 33

What are variational inference methods in Bayesian ML?

Answer 33

Optimization-based approximations that replace the true posterior with a simpler distribution chosen to minimize a divergence, often for scalability.

Question 34

Why is approximate Bayesian inference important in modern ML?

Answer 34

Exact inference is often infeasible in complex models, so practical algorithms rely on approximations to capture key posterior structure.

Question 35

What is Bayesian model averaging at a high level?

Answer 35

Combining predictions from multiple models weighted by their posterior probabilities instead of choosing a single best model.

Question 36

Why is Bayesian model averaging conceptually attractive?

Answer 36

It accounts for model uncertainty, often improving predictive performance and calibration compared to relying on one model.

Question 37

How does Bayesian thinking relate to ML hyperparameter tuning?

Answer 37

Bayesian optimization treats the performance as a function of hyperparameters and uses a probabilistic model to choose promising configurations.

Question 38

Why is a full Bayesian treatment of deep nets often impractical?

Answer 38

The parameter space is huge and highly non-linear, making exact posterior inference computationally difficult.

Question 39

What is a practical Bayesian-inspired idea used in ML without full inference?

Answer 39

Using priors as regularizers, ensembles as approximations to posterior predictive distributions, or uncertainty estimates from dropout or ensembles.

Question 40

In one sentence, what is the key mental model for Bayesian stats in ML?

Answer 40

Bayesian methods update prior beliefs with data to produce posterior distributions, which in ML often appear as regularization, uncertainty estimates, and principled predictive distributions.