Evaluating ML/DL models

Question 1

How to avoid overfitting and find the best hyper parameters’ combination?

Answer 1

Split data methods:
1. Holdout: train-validation-test data sets;
2. Cross-validation: k-fold cross validation;

Question 2

what is cross validation - k fold

Answer 2

The KCV
consists in splitting a dataset into k subsets; then, iteratively, some of them are
used to learn the model, while the others are exploited to assess its
performance.

Question 3

What is hyper parameter optimisation

Answer 3

❑ Hyperparameters influence model performance and generalisation.
❑ Examples: Learning rate, batch size, number of layers, activation functions etc.
❑ Goal: Find the best combination to maximise validation accuracy and minimise overfitting.

Question 4

What is grid search

Answer 4

❑ Exhaustive search over a manually specified set of hyperparameters.
❑ Evaluates all possible combinations systematically.

Question 5

Give an example of grid search: If learning rate = [0.01, 0.001] and optimiser = [adam, sgd]

Answer 5

Grid Search will test 2 × 2 = 4 combinations.

Question 6

What are the pros of grid search

Answer 6

❑ Systematic.
❑ Suitable for small hyper parameter spaces.

Question 7

What are the cons of grid search

Answer 7

❑ Computationally expensive as dimensions increase.
❑ Not suitable for high-dimensional or continuous spaces.

Question 8

What is random search

Answer 8

❑ Randomly samples hyperparameters from specified distributions.
❑ Number of trials is predefined.
❑ More efficient as it avoids exhaustive combinations.

Question 9

Exampe random search: With 10 trials, Random Search might sample:

Answer 9

(0.01, adam), (0.001, sgd), (0.01, sgd), …

Question 10

What are the pros of random search

Answer 10

❑ More efficient than Grid Search in high dimensions.
❑ Works well with continues hyperparameters and larger hyperparameter
space.

Question 11

What are the cons of random search

Answer 11

❑ No guarantee of finding the absolute best combination.
❑ Performance depends on the number of trials.

Question 12

Grid search vs random search vs advanced methods

Answer 12

❑ Grid Search: Thorough but expensive. Use for small parameter sets.
❑ Random Search: Efficient for large or continuous spaces.
❑ Consider advanced methods (e.g., Bayesian) for complex problems.

Question 13

What does a classification model aim to provide

Answer 13

A classification model aims to provide the
current label for each input.
* Binary classification.
* Multi – class.
* Output is commonly the probability of an
input belonging to a class.

Question 14

What is the formula for accuracy

Answer 14

Accuracy = (TP + TN)/ (TP + TN + FP + FN)

Question 15

Why is accuracy not the best classification metric

Answer 15

While 91% accuracy may seem good at first glance, another tumor-classifier model that always predicts benign would
achieve the exact same accuracy (91/100 correct predictions) on our examples.

Accuracy alone doesn’t tell the full story when you’re working with a class-imbalanced data set, like this one, where
there is a significant disparity between the number of positive and negative labels.