8.1 Model Evaluation 2

Question 1

What is model bias?

Answer 1

Model bias in classification context:

• Label predictions can’t be “too high” or “too low”
• “biased towards the majority class” means our model predicts too many instances as the majority class
• Typically compare the class distribution:
• An unbiased classifier produces labels with the same distribution as the actual distribution
• A biased classifier produces labels with a different distribution from the actual distribution

Question 2

What is model variance?

Answer 2

Model variance relates to the tendency of different training sets to produce different models or predictions with the same type of learner

• A model has high variance if a different randomly sampled training set leads to very different predictions on the evaluation set
• A model has low variance if a different randomly sampled training set leads to similar predictions, independent of whether the predictions are correct

Question 3

Which of the following is more harmful for the performance on test set than training set?

Answer 3

• Model variance

The model variance is high when different randomly sampled training sets lead to very different predictions on the test set. The high variance indicates that the model overfits to training set. In this case, the training error may decrease, but test error will increase.

Question 4

During training process, if your model shows significantly different performance across different training sets, which of the following is NOT a valid way to reduce this variance?

Answer 4

• Improve optimisation algorithm used for error minimisation

Improving your optimisation algorithm would decrease the bias. To reduce variance, using the other three options would be helpful. To decrease model complexity, you can consider reducing the number of features or using regularisations.

Question 5

Given a model y=θ₀+θ₁x , after adding more basis functions to this model, it becomes y=θ₀+θ₁x+…+θ_nxⁿ . Adding more basis functions can:

Answer 5

Adding more basis function would increase the model complexity, which would lead to decrease in model bias and increase in variance. The following figure illustrates the relationship of model complexity and model bias and variance.

Question 6

What are the possible solutions to reduce evaluation variance?

Answer 6

• Repeated random subsampling and run multiple evaluations
• Increase the holdout partition size for test set
• Use K-fold cross-validation

To reduce evaluation variance, we can increase the size of test set, or evaluate multiple times using repeated random subsampling or K-fold cross-validation, and get the average performance across different runs. The stratification generates training and test sets that contain approximately the same distribution of class labels as the overall set. The stratification can help to reduce the bias.