Model Selection

Question 1

Occams razor

Training Error / Generalization Error

Answer 1

Question 2

Statistical Learning Theory

Learning Machines

Answer 2

Question 3

Statistical Learning Theory

MSE

Answer 3

Question 4

Bias Variance Tradeoff

Answer 4

• underfitting:
  
  • high bias
  • low variance
• overfitting:
  
  • low bias
  • high variance

Question 5

James Stein paradox

Answer 5

Question 6

No free lunch theorem

Answer 6

• Occams razor (simples model / most intuitive method is preferable):
  
  • choice of model depends on scenario
• utilitaristic approach (lowest prediction error is preferable):
  
  • without knowledge of the scenario, its impiossible to predict the prediction error

Question 7

Methods for model selection

Answer 7

Question 8

k-fold cross validation

Answer 8

• never use the test set!