Deep Learning Algorithms

Question 1

Neural Networks: general

Answer 1

• Activation function: most common being ReLU with takes an input x and returns max(0, x)
• Bias prevents the neuron from being “switched off” as its added to the inputs (weights) to prevent a 0 output
• The ‘learning’ is the iterative adjustment of the bias and weights
• Initialise the NN with random weights, whereby the initial output will be very incorrect

Question 2

Neural Networks: one epoch

Answer 2

1. Forward propagation is the input and initialisation of all of our weights and biases and passing through to the output
2. Compare the output of NN with the label to see how close it was. This is done with the loss function
3. Loss function: mapping back to the network via back propagation
4. Network adjusts the weights and biases to minimise the Loss Function
5. Loss function involves Gradient Descent optimisation and tuning of the learning rate

Question 3

Convolutional Neural Network

Answer 3

• Supervised, multi-classification
• Mainly used for image classification, or objects within an image
• We have hidden “convolutional layers” within the network
• First layers are responsible for very general features of the image (i.e. broad shapes and colours)
• Further layers drill down

Question 4

CNN: filtering

Answer 4

• We apply a “convolutional filter” to the image. This filter has an existing matrix of values. We then apply this filter to a certain area of the image and carry out matrix multiplication by the values in that area. We convolve this across the whole image
• We may apply different types of filters (i.e. right-hand edges, corners etc.) These types of filters generally come within existing CNNs that we start with
• We then apply custom filters (e.g. eye, bird beak) as well. These are learned as we build the CNN

Question 5

Recurrent Neural Network

Answer 5

• Supervised, “other” output (not binary/numeric/class)
• Uses: stock predictions, time series, voice recognition (i.e. sequence to sequence)
• Often said that RNNs have “memory” in that the output of the model is passed back into the model again as the input
• Commonly used in translation, where the context of the word is very important. The “memory” would comprise the prior part of the sentence that has come before the word which the model is evaluating (model “sees” context)