Hyperparameters & Activation Functions

Question 1

What are hyperparameters?

Answer 1

Hyperparameters are configuration variables external to the model that are set before the training process begins and control how the model learns.
- define how the model is trained. how long we train, the learning rate

Question 2

What is dropout?

Answer 2

• Dropout is a regularization technique used to prevent overfitting in neural networks.
• Randomly turn off neurons

Question 3

How does dropout work?

Answer 3

During training, dropout randomly “drops out” (sets to zero) a certain percentage of neurons in a layer at each training step. This means the dropped-out neurons do not contribute to the forward pass or backpropagation for that specific training example.

Question 4

What’s the typical range of dropout?

Answer 4

0.2 to 0.5

Question 5

What is the effect of dropout?

Answer 5

• Improved generalization and reduced overfitting
• By forcing the network to learn more robust features that are less reliant on the presence of any single neuron, dropout encourages the model to learn redundant representations.
• This effectively trains an ensemble of smaller, different networks at each iteration, leading to improved generalization and reduced overfitting.

Question 6

Overfitting

Answer 6

• When a model learns the training data too well, including noise and specific patterns that do not generalize to unseen data.
• Causes the model to perform poorly on unseen data

Question 7

Learning Rate

Answer 7

- step size during the optimization process, particularly in gradient descent algorithms
- Dictates how much the model’s parameters are updated in response to the estimated error at each iteration
- Influences both the speed of convergence and final accuracy
- how fast the model parameters are allowed to change

Question 8

What happens if the learning rate is too high?

Answer 8

• the optimization process will overshoot the optimal solution
• leads to unstable training, oscillations in loss, or even divergence

Question 9

What happens if the learning rate is too low?

Answer 9

• Very slow convergence potentially getting stuck in local minima and taking a long time to reach an optimal solution

Question 10

What is a typical learning rate?

Answer 10

• Problem dependent and often requires tuning
• 0.01 is recommended starting point for general neural network training
• 0.001 for optimizers like Adam
• 0.0001 for optimizers like AdamW

Question 11

Max Depth

Answer 11

• number of layers
• control the complexity of the network
• more layers allow the model to fit more complex patterns but also increase the risk of overfitting

Question 12

Regularization Strength

Answer 12

• Add a penalty term to the model’s loss function preventing it from fitting the training data too closely and improving its ability to generalize to unseen data

Question 13

L1 Regularization

Answer 13

• Lasso
• has the effect of reducing the number of features used in the model by pushing to zero the weights of features that would otherwise have small weights.
• results in sparse models and reduces the amount of noise in the mode

Question 14

L2 regularization

Answer 14

• Ridge
• results in smaller overall weight values and stabilizes the weights when there is a high correlation between the input features

Question 15

Number of epochs

Answer 15

how many times the entire training dataset is passed through the learning algorithm

Question 16

scale_pos_weight

Answer 16

• XGBoost param
• used to address class imbalance in binary classification problems
• can adjust to reduce high FN rate

Question 17

batch_size

Answer 17

• in stochastic gradient descent, the batch size determines how many samples are used to compute the gradient at each step

Question 18

n_estimators

Answer 18

• used in random forest
• number of decision trees in the ensemble

Question 19

max tree depth

Answer 19

limits the depth of individual decision trees preventing them from becoming overly complex

Question 20

Early stopping

Answer 20

• regularization technique
• Training is stopped when the validation error starts to increase, indicating that the model is starting to overfit the training data.
• This prevents the model from continuing to learn the training data too closel

Question 21

What are some regularization techniques?

Answer 21

L1, L2, Dropout, Early stopping

Question 22

What is the goal of regularization?

Answer 22

To prevent over-fitting

Question 23

What hyerparameters should be adjusted for binary classification problems with imbalanced datasets?

Answer 23

• scale_pos_weight
• eval_metric

Question 24

eval_metric

Answer 24

• XGBoost param
• specifies the evaluation metric used to monitor the model’s performance on a validation dataset

Question 25

Underfitting

Answer 25

Performs poorly on training dataset

Question 26

What to do about underfitting?

Answer 26

- Add new domain-specific features and more feature Cartesian products - change the types of feature processing used (e.g., increasing n-grams size). - Decrease the amount of regularization used.

Question 27

Target Encoding

Answer 27

type of encoding is achieved by replacing categorical variables with just one new numerical variable and replacing each category of the categorical variable with its corresponding probability of the target.

Question 28

Standard Scaling

Answer 28

feature engineering method that transforms continuous features in a dataset by centering their mean around zero and scaling them to have a unit variance. This method helps regression models to perform better by ensuring that no single feature dominates the others due to differences in scale.