week 2

Question 1

What is the Bayes optimal classifier in binary classification?

Answer 1

The predictor f*(x) = 1[ P(Y=1|X=x) ≥ threshold ] that minimises expected risk.

Question 2

What is the likelihood ratio used in the optimal rule?

Answer 2

“ℒ(x) = ρ(x|Y=1) / ρ(x|Y=0).”

Question 3

What form does the Bayes classifier take using the likelihood ratio?

Answer 3

“f*(x)=1[ ℒ(x) ≥ (π₀(l01−l00)) / (π₁(l10−l11)) ].”

Question 4

What is a likelihood ratio test (LRT)?

Answer 4

A classifier of the form fη(x)=1[ℒ(x)≥η] with threshold η.

Question 5

What does Neyman–Pearson lemma state?

Answer 5

If class-conditional densities are continuous, the classifier that maximises TPR subject to FPR ≤ α is an LRT with threshold chosen so FPR=α.

Question 6

What are Type I and Type II errors?

Answer 6

Type I: false positive. Type II: false negative.

Question 7

What is True Positive Rate (TPR)?

Answer 7

P(f(X)=1 | Y=1).

Question 8

What is False Negative Rate (FNR)?

Answer 8

P(f(X)=0 | Y=1).

Question 9

What is False Positive Rate (FPR)?

Answer 9

P(f(X)=1 | Y=0).

Question 10

What is True Negative Rate (TNR)?

Answer 10

P(f(X)=0 | Y=0).

Question 11

How can risk be decomposed using FPR and TPR?

Answer 11

R[f] = α·FPR − β·TPR + γ for constants α,β≥0.

Question 12

What is an alternative supervised learning goal beyond minimising risk?

Answer 12

Maximising TPR subject to constraint FPR ≤ α.

Question 13

What distinguishes discriminative from generative models?

Answer 13

Discriminative models learn predictors f; generative models model ρ(x,y) or ρ(y), ρ(x|y).

Question 14

What do generative models learn?

Answer 14

ρ(x|y), ρ(y) allowing computation of ρ(y|x) via Bayes’ rule.

Question 15

What does Linear Discriminant Analysis assume?

Answer 15

Classes have Gaussian class-conditional distributions with shared covariance Σ and different means μᵢ.

Question 16

What is Quadratic Discriminant Analysis?

Answer 16

A generative model like LDA but each class has its own covariance Σᵢ.

Question 17

What assumption does Naive Bayes make?

Answer 17

Conditional independence of features xⱼ given class y.

Question 18

What loss corresponds to maximum likelihood estimation?

Answer 18

The log-loss l(x,y,θ)=−logρθ(x,y).

Question 19

What is empirical risk for maximum likelihood?

Answer 19

R̂(θ)=Σ_j −logρθ(x_j, y_j).

Question 20

Why are generative models hard in practice?

Answer 20

It’s difficult to specify a realistic density model ρ(x,y) for complex data.

Question 21

What is the purpose of regularisation?

Answer 21

To penalise model complexity and prevent overfitting by modifying the objective.

Question 22

What is explicit regularisation?

Answer 22

Adding λΩ(f) to empirical risk: J(f)=R̂(f)+λΩ(f).

Question 23

What does λ (lambda) control in regularisation?

Answer 23

The strength of the complexity penalty; it is a hyperparameter chosen with cross-validation.

Question 24

What is L2 regularisation?

Answer 24

Ω(θ)=‖θ‖₂² = Σ_j θⱼ² (ridge).

Question 25

What effect does L2 regularisation have?

Answer 25

Shrinks parameters but does not force them to zero.

Question 26

What is L1 regularisation?

Answer 26

Ω(θ)=‖θ‖₁ = Σ_j |θⱼ| (lasso).

Question 27

What effect does L1 regularisation have?

Answer 27

Encourages sparsity by driving some parameters to zero.

Question 28

How does cross-validation help choose λ?

Answer 28

Evaluate model performance on validation folds for different λ and choose the λ with best generalisation.

Question 29

What problem does Lasso regression solve?

Answer 29

Minimising (1/n) Σ (y - fθ(x))² + λ‖θ‖₁.

Question 30

What does a linear separator do?

Answer 30

Finds a hyperplane that correctly separates binary classes.

Question 31

How is a line written in vector form?

Answer 31

wᵀx − b = 0.

Question 32

How does a linear classifier assign labels?

Answer 32

Predict +1 if wᵀx − b > 0, predict −1 if wᵀx − b < 0.

Question 33

When is a point (x,y) correctly classified?

Answer 33

When y(wᵀx − b) > 0.

Question 34

What does SVM aim to maximise?

Answer 34

The margin: the distance between decision boundary and closest data point.

Question 35

What is hinge loss?

Answer 35

l(y,f(x)) = max(1 − y(wᵀx − b), 0).

Question 36

What objective does SVM minimise?

Answer 36

J(w,b)= (1/n)Σ max(1−y_j(wᵀx_j−b),0) + (λ/2)‖w‖₂².

Question 37

Why include regularisation in SVM?

Answer 37

To penalise large weights and improve generalisation.

Question 38

What is a soft-margin SVM?

Answer 38

An SVM with low C (high λ), allowing misclassified or low-margin points.

Question 39

What is a hard-margin SVM?

Answer 39

An SVM with very large C (small λ), forcing perfect separability.

Question 40

How is the perceptron related to SVM?

Answer 40

It is SVM with b=0, λ=0, trained via stochastic gradient descent.

Question 41

What probability does logistic regression model?

Answer 41

P(y=1|x) = σ(wᵀx − b).

Question 42

What loss does logistic regression typically use?

Answer 42

Cross-entropy loss l(y,z)=−y log z − (1−y) log(1−z).

Question 43

What is logistic loss for classification?

Answer 43

−1[y=1] log σ(wᵀx−b) − 1[y=0] log σ(−wᵀx+b).

Question 44

How can linear models handle non-linear boundaries?

Answer 44

By applying feature maps x → φ(x) that add nonlinear transformations.