Performance Metrics

Question 1

• How is misfit different for classification and regression

Answer 1

• Classification – you are wrong (discrete)
• Regression – how much you are wrong (continuous)

Question 2

• Type 1 vs Type 2

Answer 2

• Mistaken vs missed significance – case dependent
• false positive vs false negative

Question 3

What do the rows of the confusion matrix sum up to?

Answer 3

• Rows sum to true +ve and -ve

Question 4

• How class imbalance could cause problem for some accuracy measures.

Answer 4

• Favouring the majority thus misleading accuracy results.

Question 5

• What is the role of an accuracy measure during training of a model?

Answer 5

• Assess how well the model is learning patterns in the data

Question 6

• Accuracy measure to penalise larger misfits during model training

Answer 6

Lp norms
penalises misfits based on how wrong they are

Question 7

• What should be the AUC value for a model that makes perfect predictions?

Answer 7

1.0

Question 8

• How threshold value selection in binary model forms a value judgement? Who makes the selections?

Answer 8

• Engineer makes the judgement on where to make the split

Question 9

• 3 reasons to measure model accuracy.

Answer 9

• Credibility – stakeholder confidence
• Improvement – how, where, whether
• Comparison – assist with model or feature selection