Model Evaluation (2)

Question 1

Confusion Matrix

What types of predictions is a Confusion Matrix used to measure?

Answer 1

Applies to discrete answers, like Classification models (not continuous values)

Question 2

Confusion Matrix

What is the Accuracy of a model?

Answer 2

The % of predictions that are correct (true positive and true negative)

Question 3

Confusion Matrix

What is the Precision of a model?

Answer 3

How well does a model predict

Question 4

Confusion Matrix

What’s the ratio considered for Precision?

Answer 4

true positives vs. false positives

Question 5

Confusion Matrix

Formula for Precision?

Answer 5

true positives / (true positives + false positives)

Not important

Question 6

Confusion Matrix

What is invariant about Precision?

Answer 6

The model said ‘yes’, decide if it was right or not.

Question 7

Confusion Matrix

What can you minimize by paying attention to Precision?

Answer 7

Minimize the number of false positives

Question 8

Confusion Matrix

Example of when high Precision is important?

Answer 8

Saying the defendant is guilty

Question 9

Confusion Matrix

What is the Recall of a model?

Answer 9

Does the model say ‘yes’ when it should say ‘yes’?

Question 10

Confusion Matrix

What’s the ratio considered for Recall?

Answer 10

True positives vs. false negatives

Question 11

Confusion Matrix

Formula for Recall?

Answer 11

True positives / (true positives + false negatives)

Not important

Question 12

Confusion Matrix

What is the invariant about Recall?

Answer 12

The actual value is ‘yes’. What did the model say?

Question 13

Confusion Matrix

What can you minimize by paying attention to Recall?

Answer 13

The number of false negatives (times the model should have said ‘yes’, but didn’t)

Question 14

Confusion Matrix

Example of when high Recall is important?

Answer 14

Predicting if you have cancer

Question 15

Confusion Matrix

How can you optimize a model for both Precision and Recall?

Answer 15

You can’t, try to balance the model based on your needs.

Question 16

Confusion Matrix

How do you measure if your model is balanced?

Answer 16

“F1” is a metric that balances Precision and Recall when both are important.

Question 17

Confusion Matrix

What’s the formula for F1?

Answer 17

F1 = (precition * recall * 2) / (precision + recall)

Not important

Question 18

Confusion Matrix

What is Model Accuracy?

Answer 18

How well the model guesses correctly

Question 19

Confusion Matrix

Quantitatively, what is Accuracy?

Answer 19

Amount of true positives and true negatives compared to all answers

Question 20

Confusion Matrix

What metric is best when false positives are costly?

Answer 20

Precision

Question 21

Confusion Matrix

What metric is best when false negatives are costly?

Answer 21

Recall

Question 22

Errors in regression

What does MAE stand for?

Answer 22

Mean Absolute Error

Question 23

Errors in regression

What is Mean Absolute Error?

Answer 23

Distance from data-point to regression line

Question 24

Errors in regression

What does MAPE stand for?

Answer 24

Mean Absolute Percentage Error

Question 25

# *Errors in regression* What is RMSE?

Answer 25

Root mean square error

Question 26

# *Errors in regression* What is RMSE used for?

Answer 26

Tries to smooth-out the variance between errors

Question 27

# *Errors in regression* What is "R squared"?

Answer 27

Explains variance in your model

Question 28

# *Errors in regression* What does "R squared" near 1 mean?

Answer 28

The predictions of your model are good

Question 29

# *Errors in regression* Which of MAE, MAPE, RMSE, and R^2 are used for continuous values vs. discrete?

Answer 29

All are for continuous only (curve-fitting)

Question 30

# *Errors in regression* What does "model has RMSE of 5" mean?

Answer 30

On average, the model predicts a value 5 points off of the actual value