Loss Functions

Question 1

Loss functions

Answer 1

Loss functions are a crucial part of machine learning algorithms. They measure the inconsistency between predicted and actual outcomes and guide the learning process to adjust the model parameters.

Question 2

1. Mean Squared Error (MSE)

Answer 2

Used for regression tasks, MSE calculates the average squared difference between the actual and predicted values. This puts a high penalty on large errors. However, it’s sensitive to outliers since errors are squared before they’re averaged.

Question 3

2. Mean Absolute Error (MAE)

Answer 3

Also used for regression, MAE calculates the average absolute difference between actual and predicted values. It is less sensitive to outliers compared to MSE and provides a linear penalty to errors.

Question 4

3. Binary Cross-Entropy (Log Loss)

Answer 4

Used in binary classification problems, it calculates the log of the likelihood of the true label given the predicted probability. It has the benefit of punishing the model heavily when it is confident and wrong.

Question 5

4. Categorical Cross-Entropy

Answer 5

Used in multi-class classification problems, it is a generalization of Binary Cross-Entropy. It calculates the negative log likelihood of the true label given the predicted probability distribution over all classes.

Question 6

5. Hinge Loss

Answer 6

Used in Support Vector Machines and some neural network classifiers. It penalizes misclassified examples and disregards correctly classified ones, maximizing the margin of the decision boundary.

Question 7

6. Kullback-Leibler (KL) Divergence

Answer 7

It’s used when we want to compare two probability distributions. It is often used in unsupervised learning algorithms.

Question 8

7. Huber Loss

Answer 8

A combination of MSE and MAE. It behaves like MSE for small errors and MAE for large errors. It is less sensitive to outliers than MSE and is often used in robust regression.

Question 9

8. Quantile Loss

Answer 9

Used in quantile regression, for predicting an interval instead of a single point. It penalizes underestimates more heavily for higher quantiles and overestimates more heavily for lower quantiles.

Question 10

9. Focal Loss

Answer 10

An adaptation of the binary cross entropy loss, designed to address class imbalance by assigning more weights to hard-to-classify instances.