Deep Learning Basics

Question 1

1. What happens if you apply a 5x5 filter to a 7x7 image with no padding, stride = 1.
2. What happens if you apply a 5x5 filter to a 7x7 image with no padding, stride = 2.
3. What happens if you apply a 5x5 filter to a 7x7 image with no padding, stride = 3.

Answer 1

1. Get a 3x3
2. Get a 2x2
3. Not possible as the filter would go outside the image

Question 2

What is stride?

Answer 2

Stride is how many pixels along we move the filter each time.
Stride = 1, means we move 1 pixel in any direction.

Question 3

What is the formula for the output size of an image?

What is the size of the following:
1. 7x7 image, filter = 3x3, stride = 1
2. 7x7 image, filter = 3x3, stride = 2
3. 7x7 image, filter = 3x3, stride = 3

Answer 3

((N-F)/Stride) + 1

stride 1 => ((7 - 3) / 1) + 1 = 5
stride 2 => ((7 - 3) / 2) + 1 = 3
stride 3 => ((7 - 3) / 3) + 1 = 2.33 - not recommended because it’s not an integer value.

Question 4

What is the motivation of padding?

Answer 4

To obtain an output size that is the same as the input image size.

Question 5

What formula do we use to calculate how the size we would pad with?

How much should we pad for filter size:
- 3
- 5
- 7

Answer 5

(F-1)/2

(3-1)/2 = 1 (padding)
(5-1)/2 = 2 (padding)
(7-1)/2 = 3 (padding)

Question 6

How do we deal with images that have a depth of 3 - RGB images (larger than 1)

Answer 6

We must use a filter with a depth that matches in input image depth. E.g. for an RGB image of depth 3, the filter must have depth 3.
- We calculate the dot product to merge the 3 output values into 1. Perform the convolution then add the 3 values to get 1.

Question 7

What happens if the image depth and filter depth aren’t the same value?

Answer 7

We can’t calculate their dot product.

Question 8

We may want to use multiple filters, how does this effect the output size?

Answer 8

The filter size decides the depth of the output, the output is known as the number of activation maps

Question 9

Convolutional Neural Networks (CNN) size
In one convolution layer, we have 128 filters of 3x3x3 applied to input volume 128x128x3 with stride 2 and pad 1. What is the size of the output volume? Give details of how you calculate the size of the output volume.

Answer 9

(((N - F + (2*P))/stride) + 1

(((128-3+(2*1))/2)+1 = 64.5

round down/floor to 64

so output is 64x64x128
- number of filters is always the same as the number of output activation maps

Question 10

What are the formulas for calculating the output images height, width and depth:

Answer 10

a volume of size W1 x H1 x D1
Four hyperparameters are required: Number of filters K, Filter size F, stride S, amount of zero padding P

When W2 and H2 are integers:
* Next layer: a volume of size W2 x H2 x D2
W2 = (W1 - F +2P) / S +1
H2 = (H1 - F +2P) / S +1
D2 = K

When and are not integers:
Next layer: a volume of size W2 x H2 x D2
▪ W2 = floor((W1- F +2P) / S) +1
▪ H2 = floor((H1 - F +2P) / S) +1
▪ D2 = K

Question 11

Explain the role of the pooling layer in CNN:

Answer 11

• performs down-sampling making image representations smaller and more manageable, by aggregating information, reducing computation costs and memory usage
• e.g. given an input image of 200x200, it can make it 100x100.
• it operates over each activation map independently. The width and height get smaller but the depth remains the same

Question 12

List two methods of down-sampling/pooling layers:

Answer 12

Max pooling: given a subregion, take the max value. The next region is where the next stride is. max(2, 9, 4, 5) = 9

Average pooling: given a subregion, calculate the average of the pixels. Next region is where the next stride is. avg(2, 9, 4, 5) = 20/4 = 5

Question 13

What is a drawback of max and average pooling:

Answer 13

• They may remove important information or whole features from an image when down-sampling.

Question 14

Calculate the output size of the pooling layer:

Give the output size for this question:
image = 128x128x3, Filter/pooling size = 2x2, stride = 2, no padding

Answer 14

image: W x H x D, filter F, stride S
- depth is the same
- W = floor((W - F)/S) + 1
- H = floor((H - F)/S) + 1

D = 3
W = ((128-2)/2)+1 = 64
H = ((128-2)/2)+1 = 64
output = 64x64x3

Question 15

Explain the role of the fully connected layer in CNN:

Answer 15

• connects all neurons, so every input of the input vector influences every output of the output vector.
• maps learned high level features to the final output

Question 16

Explain the process of the fully connected layer:

What is the output size of a fully connected layer with input image 32x32x3, 10 categories.

How many parameters would there be:

Answer 16

step 1) stretch channels into a 1D vector
step 2) connect each input node to each output node
step 3) y = Wx (ignoring bias)

• the 32x32x3 image becomes a 3072x1 vector
• the weights matrix would consist of 10x3072, perform matrix multiplication to get a 10x1 output

there would be 30720 parameters without bias. (30730 with bias)

Question 17

What is the role of the activation function in CNN:

Answer 17

• it introduces non-linearity to the network, enabling it to learn more complex relationships/patterns in the data.
• They can mitigate vanishing gradients

Question 18

What is the role of the BatchNorm layer in CNN?

Answer 18

• It normalises the output of each layer in the network
• Ensures outputs of a layer have mean = 0 and standard deviation = 1 which improves training stability
• Improves convergence speed. Can use random initialisation and a big learning rate

Question 19

When is the BatchNorm layer applied:

Answer 19

After the convolution layer but before the activation function

Question 20

What happens if we don’t perform batchNorm:

Answer 20

The state distributions are very strange for each layer, they are unknown and vary for each iteration.

Question 21

Why do we forward batches through model instead of single images:

Answer 21

This creates a more stable network.

Question 22

Why does batchNorm have two learnable parameters?

Answer 22

• γ (scale) and β (shift) for each channel
  -The parameters γ and β allow the network to learn the optimal scale and shift for each feature map.

Question 23

Why is the training process more stable using batchNorm?

Answer 23

Because the moving average of the mean and standard deviation is used to update them.

Question 24

What are the differences between a standard network and a network that uses batchNorm?

Answer 24

• standard has to initalise parameters beforehand, have to design careful initialisation strategy
• BatchNorm can use random initialisation
• standard sometimes can’t use a big learning rate, converges slower
• batchNorm uses a big learning rate so converges quicker
• batchNorm uses many more layers than standard

Question 25

What affect does batchNorm have on the number of epochs required to train a deep network?

Answer 25

- it dramatically reduces the number of epochs, as it can use a bigger learning rate and therefore converge faster

Question 26

What is the role of the dropout layer in CNN:

Answer 26

- Dropout is used to overcome overfitting to the training data and improve generalisation of the model

Question 27

When might overfitting take place?

Answer 27

If there's a relatively large number of parameters with an insufficient number of training samples, overfitting often occurs

Question 28

How is the dropout operation/layer performed?

Answer 28

- For each node we calculate a probability of dropout. - if we set a dropout rate at 30%, then 30% of the nodes in that layer will be dropped - if a node is dropped all the connections in the higher layers will be disconnected.

Question 29

Why may we only perform dropout in the later layers of our neural network:

Answer 29

- early dropout may disrupt the gradient - perform dropout only on deeper layers to preserve feature extraction in the early layers - deeper layers are more prone to overfitting so more effective to perform dropout then

Question 30

What is the role of the ReLu layer?

Answer 30

- an activation function, that improves the non-linear representation ability of the network.

Question 31

What is the role of the deconvolutional layer?

Answer 31

- aka transposed convolution - used to upsample feature maps to a bigger size - useful when you need the output to match the size of the input dimensions - used for semantic segmentation and used by GANs for image generation,

Question 32

What does regularlisation achieve?

Answer 32

- prevent overfitting - improves model generalisation - stabilise the training process - reduce model complexity

Question 33

What can happen if your dataset is too small?

Answer 33

- underfitting, due to bias

Question 34

What technique can be used if the dataset is too small?

Answer 34

N-Fold Cross Validation

Question 35

What is N-Fold Cross Validation?

Answer 35

- split the data into n folds - train model on n-1 folds and 1 fold for testing. - each round a different fold is used as the test set, while the rest are used for training - after each round the model's performance is recorded - for n folds you get n performance scores - these are then averaged to get the final model evaluation score

Question 36

What is the long tail problem? and what is the solution to the tail problem?

Answer 36

- Many categories but the dataset has few examples of each category, so it’s hard to train the model accurately. - randomly splitting the dataset could mean the training dataset doesn't even include one instance of a category. All the instances will be in the validation set, so the model will have no ability to predict this category. - solution is n-fold cross validation, as this makes sure every image is in the training and test datasets at least once.

Question 37

What evaluation metrics can we use for a model performing binary classification?

Answer 37

Two categories, positive (1) and negative (0) Classifier predicts: TP - true positives: number of positive samples that the classifier has correctly identified TN - true negatives: number of negative samples that the classifier has correctly identified 'not lion' FP - false positives: number of positive samples that the classifier has incorrectly identified FN - false negatives: number of negative samples that the classifier has incorrectly identified

Question 38

Given the metrics used to evaluate a classifiers performance on binary classification, what are the formulas we can calculate with them?

Answer 38

Precision: TP/TP+FP Recall: TP/TP+FN F1: 2* (precision*recall)/precision+recall -combines precision and recall to get the final performance of the model Accuracy: TP+TN/TP+TN+FP+FN - correct predictions over total dataset

Question 39

What qualities lead to underfitting, fitting well and overfitting:

Answer 39

underfitting: high bias fit well: low bias, low variance overfitting: high variance

Question 40

What can we do to fix underfitting:

Answer 40

- train the network with more epochs

Question 41

What can we do to fix overfitting:

Answer 41

- data augmentation - add regularlisation layers

Question 42

What are L1 and L2 regularlisation methods?

Answer 42

- used to prevent overfitting - by adding a penalty term to the loss function to encourage smaller weights

Question 43

What is the role of the convolution layer in CNN?

Answer 43

- It extracts local features by applying convolutional filters/kernels to output feature maps.

Question 44

What are the differences between the training, validation and test datasets? How is a dataset split into training, validation and testing?

Answer 44

* Training set: Training models * Validation set: Test the model trained on training set and select the best performed model * Testing set: Test the model * Training set vs validation set: 80% vs 20% or 90% vs 10% * Training vs validation vs testing set: 60%-20%-20% * No standard rule, if a large amount of data, e.g., 1,000,000. It is possible to have 98%-1%-1% * Sometimes we only get the training & validation set (in some competitions the test set isn’t available to us)

Question 45

How is L1 regularlisation calculated?

Answer 45

loss = squared error loss + penalty term loss = sum of (actual - input*weight)^2 + sum of absolute value of the weights

Question 46

How is L2 regularlisation calculated?

Answer 46

loss = squared error loss + penalty term loss = sum of (actual - input*weight)^2 + sum of (weights^2)

Question 47

What is the difference between L1 and L2 regularlisation loss:

Answer 47

L1: the penalty term added is proportional to the absolute values of the weights L2: the penalty term added is proportional to the square of the weights L1 loss encourages sparsity in the weights L2 loss discourages large weights

Question 48

What are some regularlisation techniques?

Answer 48

- L1 and L2 regularlisation - Dropout - early stopping - N-Folds cross validation

Question 49

Why do dropout layers help with regularlisation?

Answer 49

Ensures the model doesn't put too much weight on a single node, as it may get dropped

Question 50

When training a deep network, what could be the reason why the loss doesn't decrease in a few epochs?

Answer 50

- The learning rate is too slow - The model is overfitted

Question 51

How is early stopping used as a regularlisation strategy:

Answer 51

Early stopping stops the training process when validation loss is small

Question 52

What is an epoch?

Answer 52

The period of time/amount for an entire dataset to be passed forwards and backwards through a NN.

Question 53

Calculate the epoch size: Total samples: 1000 Batch size: 50

Answer 53

1000/50 = 20 epoch size = 20 batches or iterations

Question 54

What are the benefits of data augmentation

Answer 54

- Prevents models from overfitting - used when the initial training set is too small - improves model accuracy as improves model generalisation

Question 55

Give some examples of data augmentation:

Answer 55

- Image Flip - Image crop - image rotation - CutMix

Question 56

Describe the network training pipeline:

Answer 56

1) given a training dataset, of images and the ground truth, design a neural network, a CNN 2) perform the forward passes, which extracts the image features and produces predictions for each of the categories. 3) choose the max value of the predictions as the category of the image 4) use the ground truth to calculate the loss and then perform backpropagation to update the parameters 5) train the model, iteration by iteration

Question 57

Describe one iteration of the training process:

Answer 57

1) sample a batch of the dataset (e.g. 25 images) 2) forward propagate the batch through the network to calculate the loss 3) backpropagate to calculate the gradients 4) update the parameters using the gradient`

Question 58

How does batchNorm normalise data:

Answer 58

Transforms the data to have a mean of 0 and a standard deviation of 1

Question 59

What is a mini-batch:

Answer 59

Used when our batch size is very large and we can't forward all the data through in one iteration, so this is a sample of the data.

Question 60

If the training data has 32,000 images, and batch size is set to 32, how many mini batches will there be?

Answer 60

1000 mini-batches

Question 61

Describe the softmax loss function: Give the formula:

Answer 61

- normalise data to be in range [0,1], as we want the predictions for the categories - all predictions add up to 1 Zj is the logit/raw score for class j e^zj = numerator K = number of categories the sum of all e^k = demonimator softmax for each category = e^zj/sum of (e^k) the labelled category is the max value of the softmax predictions

Question 62

What does softmax loss do?

Answer 62

It converts raw logits (output scores from a NN) into probabilities.

Question 63

What does cross entropy loss measure?

Answer 63

It measure how well the predicted probability distribution aligns with the true labels.

Question 64

When is the cross entropy loss function applied?

Answer 64

After the softmax loss function

Question 65

What is the cross entropy loss formula?

Answer 65

-1/n * for each n (sum of ground truth*log(softmax prediction) where n is the number of samples for 1 sample with ground truth [1 0 0] and prediction [0.66 0.24 0.1]: -1/1*(1*log(0.66)+0*log(0.24)+0*log(0.1)) = -log(0.24)

Question 66

What is the formula for the MSE mean squared error loss function:

Answer 66

MSE = 1/n*sum of (actual - prediction)^2

Question 67

What does a model optimiser do?

Answer 67

Adjust the model's parameters during training to minimise loss.

Question 68

What does the learning rate parameter do?

Answer 68

- multiply learning rate by gradient (calculated using chain rule) to update weights - It determines how fast or slow we move towards the optimal weights.

Question 69

What is the difference between calculating gradient descent, stochastic gradient descent and mini-batch gradient descent:

Answer 69

- gradient descent: pass all dataset through model, then calculate loss and use that to get the gradient - stochastic: pass 1 image through, calculate loss and use this to get the gradient - mini-batch: pass mini-batch through, calculate loss for this and use this to then get the gradient

Question 70

What will a small learning rate result in?

Answer 70

- updates to weights are small, may take too long to converge or get stuck at local minima

Question 71

What will a large learning rate result in?

Answer 71

- model learns fast, may cause the model to oscillate around or jump over minima. Might cause weights to overflow/not converge

Question 72

What does a good learning rate look like:

Answer 72

Tradeoff between convergence rate and overshooting. - not too small so that our algorithm can converge swiftly - not too large so that our algorithm won’t jump back and forth without reaching an undesirable local minima.

Question 73

What is learning rate decay?

Answer 73

- minimise learning rate over time - It decreases as the number of epochs increases - can either have multi-step LR decay or exponential decay