Statistical Learning

Question 1

Difference between prediction and inference?

Answer 1

minimize prediction error (ŷ)
create interpretable model (f(x))

Question 2

What is a parametric approach?

Answer 2

1. make assumption about the functional form
2. find procedure to train model of that form

Question 3

What is supervised learning?

Answer 3

for each observation of a set of features Xᵢ there is a measured response yᵢ

Question 4

Difference regression- and classification problem?

Answer 4

r: quantitative response
c: qualitative response

Question 5

What is the variance of a model?

Answer 5

how much f(x) changes with different sets of training data

Question 6

What is the bias of a model

Answer 6

the error introduced by approximation and simplification

Question 7

What is the error rate?

Answer 7

incorrect predictions / total number of predictions
​

Question 8

What is the accuracy?

Answer 8

1 - error rate

Question 9

What is the bayes-classifier?

Answer 9

the classifier with the smallest probability of misclassification given the same set of predictors (benchmark)

Question 10

K-nearest neighbours classification:

Answer 10

identify k-nearest neighbours, then estimate conditional probability for class j, assign x to class with highest probability

Question 11

Residual Sum off Squares (RSS)

Answer 11

RSS=1∑n​ (yᵢ−ŷ​ᵢ​)²

Question 12

Standard Error (SE(ß))

Answer 12

SE(ß) = σ² / 1∑n​ (xᵢ - x̄)²

Question 13

F-test formula

Answer 13

F = ((TSS - RSS) / p) / RSS / (n-p-1)

if F > F crit reject H0

Question 14

Forward Selection:

Answer 14

starting from null model, the variable associated with the lowest RSS for its linear model is added, this continues until a stopping rule is satisfied

Question 15

Backward Selection

Answer 15

Starting with a full model, step by step the variable with the largest p-value is removed until a stopping condition is reached

Question 16

Mixed Selection:

Answer 16

like forward selection but if at any point a p-value exceeds a threshold the corresponding variable is removed, this procedure continues until each variable is either inside the model or all variables outside the model were discarded

Question 17

Polynomial regression

Answer 17

Y = ß0 + ßvariable + ßvariable² …
in R: y ~ poly(variable, n)
with n being the highest order polynomial

Question 18

Heteroscedasticity

Answer 18

the variance of the response variable is not constant
εᵢ is dependent on yᵢ

Question 19

standardization

Answer 19

rescale data to have mean 0 and standard deviation 1.
xnew = (xᵢ – x̄) / s
s = standard deviation

Question 20

normalization

Answer 20

rescaling the data so that every observation falls between 0 an 1
xnew = (xi – xmin) / (xmax – xmin)

Question 21

Outlier

Answer 21

observation with unusual and significantly different response yᵢ

Question 22

High leverage point

Answer 22

observation with an unusual set of features that has “more weight” in determining the model due to its distance from other observations

Question 23

Variance inflation factor

Answer 23

measures collinearity: 1 / 1 - R²
where R² is the R² of a regression from Xj onto all other predictors
1 = no multicollinearity, >5 = problematic

Question 24

Logistic function

Answer 24

p(x) = e^f(x) / 1 + e^f(x)
p(x) is between 0 and 1 -> probability

Question 25

Odds

Answer 25

p(x) / 1 - p(x) increasing X by one unit changes the log odds by ß1

Question 26

What is Likelihood?

Answer 26

Likelihood measures how plausible different parameter values are, given the observed data. It’s a function of parameters for fixed data. In contrast, probability is a function of data for fixed parameters.

Question 27

What does linear discriminant analysis do?

Answer 27

linear discriminant analysis considers two criteria simultaneously : maxmizing distance between class means u and minimizing class variance (scatter = s²) while its performing complexity reduction (u1 - u2)² / s1² + s2² -> ideally large

Question 28

What is sensitivity? (aka. recall)

Answer 28

true positives / all positives (TP + FN)

Question 29

What is specificity?

Answer 29

true negatives / all negatives (TN + FP)

Question 30

What is Precision?

Answer 30

true positives / all predicted positives (TP + FP)

Question 31

What is Type 1 Error?

Answer 31

False positive rate = false positives / all negatives (False Alarm)

Question 32

What is Type 2 Error?

Answer 32

False negative rate = false negatives / all positives (miss rate)

Question 33

What assumptions does LDA make?

Answer 33

fk is the density function for a multivariate normal random variable with class specific mean uk and shared covariance matrix ∑

Question 34

What assumptions does QDA make?

Answer 34

fk is the density function for a multivariate normal random variable with class specific mean uk and class specific covariance matrix ∑k

Question 35

What assumption does Naive Bayes make?

Answer 35

within the kth class, the p predictors are independent

Question 36

When to use LDA over QDA and vice versa?

Answer 36

QDA estimates far more parameters (Kp(p+1)/2 versus Kp) so LDA has less variance. In contrast if LDA assumption is off it suffers from high bias. If n is large relatively to p then QDA is preferred as variance is less of a concern

Question 37

What is a poisson distribution?

Answer 37

y takes non-negative values as its independent events that happen in x time/distance/space. Where n is extremely large while the probability is infinitely small

Question 38

Cross validation

Answer 38

split data into k folds, then use 1 fold for validation of model trained on all other folds.

Question 39

Bootstrapping

Answer 39

real population -> sample -> bootstrap samples from a sample draw new samples with replacement to estimate the distribution, variance and confidence interval for any statistic

Question 40

Best subset selection

Answer 40

1.fit all (p over k) models that contain exactly k of the p predictors 2.Pick the best model from among them and call it Mk 3.Select a single best model from M0,Mk...Mp using cross validated prediction error or adjusted R2

Question 41

What to minimize in a ridge regression?

Answer 41

RSS + λ (p∑j=1 ßj²) where λ >= 0 is a tuning parameter that penalizes big coefficients. So ridge regression shrinks less significant coefficients to 0

Question 42

What to minimize in lasso regression?

Answer 42

RSS + λ (p∑j=1 |ßj|) where λ >= 0 is a tuning parameter that penalizes big coefficients. can shrink coefficients to exactly zero