L5: Resampling Methods

Question 1

Explain the cross validation approach

Answer 1

1. Split the total data set into train/test sets. E.g. 70/30 split
2. Train the model on the train set
3. Validate the model’s accuracy/performance using the test set

Question 2

What are some drawbacks of CV?

Answer 2

1. The validation estimate of the test error rate can be highly variable, as this is generally a smaller dataset.
2. The model will be trained on fewer data points and hence perform worse than if it were trained on the whole dataset.

Question 3

Explain Leave-One-Out Cross-Validation

Answer 3

This is where the test data set consists of only one data point.

We train the data on the n-1 data set and repeat this n times, until all data points have been used as the validation set.

The LOOCV estimate for the test MSE is then the average of all test errors.

Question 4

Explain k-fold CV

Answer 4

This involves splitting the dataset into K subsets, and utilising only one of the subsets at the test dataset.

Then this is repeated K times until all K subsets have been used as the test.

The test error is averaged from the K MSE estimates.

K is typically 5 or 10

Question 5

Where is the best model complexity, ABC, based on the train and testing error? Why?

Answer 5

A. We want the testing error to be as low as possible. This is where the model is generalised well to new unseen data.

Question 6

What is the bootstrapping method?

Answer 6

1. A sample from population with sample size n.
2. Draw a sample from the original sample data with replacement with size n, and replicate B times, each re-sampled sample is called a Bootstrap Sample, and there will totally B Bootstrap Samples.
3. Evaluate the statistic of θ for each Bootstrap Sample, and there will be totally B estimates of θ.
4. Construct a sampling distribution with these B Bootstrap statistics and use it to make further statistical inference, such as:

• Estimating the standard error of statistic for θ.
• Obtaining a Confidence Interval for θ.