FCNs

Question 1

What is the point of activation functions

Answer 1

Without activation functions, a neural network would only perform linear transformations, no matter how many layers it has. Activation functions make the network non-linear, which allows it to model real-world problems such as image recognition and language understanding.

Question 2

List all the non-linear activation function

Answer 2

Sigmoid
Softmax
Tanh
ReLU
Leaky ReLU

Question 3

Explain the sigmoid activation function

Answer 3

The sigmoid activation function maps inputs to values between 0 and 1, allowing outputs to be interpreted as probabilities. However, it suffers from the vanishing gradient problem. When the input is very large or very small weight updates become extremely small which slows learning

Question 4

Explain the softmax function

Answer 4

The softmax function takes a vector of raw scores (also called logits) and converts them into a probability distribution over multiple classes.

Question 5

give an example of softmax

Answer 5

The network produces raw scores for each class
Example: [2.0, 1.0, 0.1]
Softmax:
Makes all values positive
Emphasises larger scores
Normalises them so they sum to 1
Output becomes probabilities:
Example: [0.7, 0.2, 0.1]
The class with the highest probability is the prediction.

Question 6

What is tanh activation function

Answer 6

The tanh (hyperbolic tangent) activation function maps any real input to a value between −1 and 1.

Question 7

Why is tanh useful

Answer 7

Outputs include negative, zero, and positive values

This makes it easier for the network to represent:
- Positive relationships
- Negative relationships
Being zero-centred helps training converge faster than sigmoid

Question 8

What problem does tanh also suffer

Answer 8

the vanishing gradient problem

Question 9

What is the reLU activation function

Answer 9

ReLU (Rectified Linear Unit) outputs the input directly if it is positive, and 0 if it is negative.

Question 10

Where is reLU used

Answer 10

It is widely used in hidden layers because it is computationally efficient and reduces the vanishing gradient problem, although it can suffer from the dying ReLU issue.

Question 11

what is back propagation

Answer 11

An algorithm that computes gradients of the loss with respect to each weight using the chain rule.

Question 12

Why is back propagation essential

Answer 12

Because it allows the network to learn by identifying how each parameter affects the error.

Question 13

What is the learning rate

Answer 13

The learning rate controls the step size of weight updates during optimisation.

Question 14

What is the purpose of a loss function

Answer 14

The “loss function” is a function that measures how badly the AI system did by comparing its predicted output to the ground truth and then penalising the model

Question 15

Explain why cross-entropy is preferred over squared error for classification.

Answer 15

Because cross-entropy heavily penalises confident incorrect predictions and aligns with probabilistic outputs.

Question 16

Why is binary cross-entropy paired with sigmoid activation?

Answer 16

Because sigmoid outputs probabilities in the range [0-1]

Question 17

What does minimising cross-entrophy result in

Answer 17

maximising liklihood

The model becomes more confident when it is correct
And is heavily penalised when it is confidently wrong

Cross-entropy penalises the model if it gives high probability to a wrong class — especially if it’s very confident and wrong.

Question 18

In deep learning, an activation function should be

Answer 18

1. Differentiable
2. A function of its input (the neuron’s pre-activated value)
3. a smooth continuous function

Question 19

Derivative/gradient

Answer 19

• gradient is the generalisation of a derivative
• Gradient indicates the direction and rate of the steepest ascent of the error function
• Since we want to minimise the error function, we move in the opposite direction

Question 20

What activation function is typically paired with binary cross-entropy?

Answer 20

,Sigmoid.

Question 21

What happens in binary cross-entropy when y = 1?,

Answer 21

The loss penalises low predicted probability y*.

Question 22

What happens in binary cross-entropy when y = 0?

Answer 22

The loss penalises high predicted probability y*.

Question 23

What activation function is paired with multi-class cross-entropy?

Answer 23

Softmax

Question 24

How do you find the best weights that yield the smallest loss

Answer 24

we could solve gradient(loss)(W) = 0, we’d directly find the best weights.

Question 25

Why can't we find the best weights

Answer 25

Why don’t we actually solve it? - Neural networks are too complex - Loss functions are non-linear - Millions of weights So instead, we approximate the solution using gradient descent.

Question 26

Explain Stochastic gradient descent

Answer 26

* Select a random training sample xi and corresponding target yi. * Run the network on xi to obtain prediction y_predi. * Compute the loss of the network on the sample xi, a measure of the mismatch between y_predi and yi. * Compute the gradient of the loss with regard to the network’s parameters (a backward pass). * Move the parameters a little in the opposite direction from the gradient—for example W -= step * gradient— thus reducing the loss on the sample a bit.

Question 27

Explain mini batch stochastic gradient descent

Answer 27

* Draw a batch of training samples x and corresponding targets y. * Run the network on x to obtain predictions y_pred. * Compute the loss of the network on the batch, a measure of the mismatch between y_pred and y. * Compute the gradient of the loss with regard to the network’s parameters (a backward pass). * Move the parameters a little in the opposite direction from the gradient—for example W -= step * gradient—thus reducing the loss on the batch a bit.

Question 28

How can dropout overcome overfitting

Answer 28

During training, dropout randomly “drops” (turns off) a fraction of neurons in each layer. This forces the network to learn redundant and more robust features, rather than memorising the training data. As a result, the model generalises better to unseen data as it focuses on more general features.

Question 29

What are hyper parameters that can be tuned

Answer 29

For a given neural network, there are several multiple parameters that can be optimised including the number of hidden neurons, BATCH_SIZE, number of epochs.

Question 30

What is the point of hyper parameter tuning

Answer 30

Hyperparameter tuning is the process of finding the optimal combination of those parameters that minimise the loss function.

Question 31

What does Epochs tell us

Answer 31

it tells us the number of times all the training samples have passed through the neural network during the training process.

Question 32

What does Batch size tell us

Answer 32

It is used to partition the training data in mini batches to pass them through the network. In Keras, the batch_size is the argument that indicates the size of these batches that will be used in the fit() method in an iteration of the training to update the gradient.

Question 33

What does learning rate do

Answer 33

Gradient descent algorithms multiply the magnitude of the gradient by a scalar known as learning rate (also sometimes called step size) to determine the next point.

Question 34

What is one of the commonly used learning optimisers

Answer 34

Adam (adaptive moment estimation) - controls the step size in the model