Cross Validation

Question 1

• What cross in cross validation mean

Answer 1

• Learning from training data to map across to test

Question 2

• 2 ways dataset could be split into train and test sets

Answer 2

either part of the domain (space or time)

Question 3

• Pseudoprospective vs prospective forecasting

Answer 3

• Pseudo – Mimics a prospective but uses historical data from past events (data not trained on)
• Prospective – collect data in present time and use that as testing data

Question 4

• Why hindcasts are unreliable in measuring predictive skill of model?

Answer 4

• May have data leakage

Question 5

• How out-of-sample testing assist with hyperparameter selection

Answer 5

which hyperparameter is better at generalising

Question 6

• What features of data create challenges for model generalization

Answer 6

• Imbalanced data
• Outliers
• noise

Question 7

• How to watch out for information leakage

Answer 7

• Perform data cleaning

Question 8

• What features of models create challenges for their generalization?

Answer 8

• A model that is too good fit to training

Question 9

• Why pipeline is good for describing ML workflow

Answer 9

• It is a procedure

Question 10

• In pipeline, what steps can bias enter into the analysis?

Answer 10

• Every step