week 6

Question 1

What is a deep neural network (DNN)?

Answer 1

A composition of layers fθ = fL ∘ fL−1 ∘ … ∘ f1 that maps input x through multiple transformations.

Question 2

What is the forward pass through layer l?

Answer 2

zl+1 = σl(Wl zl + bl).

Question 3

What is a shallow neural network?

Answer 3

A neural network with one hidden layer: fθ(x)=ηᵀ σ(Wx+b).

Question 4

What are W and b in a neural network layer?

Answer 4

W are weights and b are biases for that layer.

Question 5

What are activations?

Answer 5

The outputs after applying the activation function: zl = σ(Wl−1 zl−1 + bl−1).

Question 6

What are pre-activations?

Answer 6

The linear part before the activation function: al = Wl−1 zl−1 + bl−1.

Question 7

What is a neuron?

Answer 7

A single hidden unit computing σ(wᵀx + b).

Question 8

What is network depth?

Answer 8

Number of hidden layers (sometimes +1).

Question 9

What is network width?

Answer 9

Number of neurons in a layer.

Question 10

What is network capacity?

Answer 10

Total number of neurons across the network.

Question 11

What does a sigmoid activation do?

Answer 11

σ(u) = 1 / (1 + e^(−u)).

Question 12

What is the ReLU activation?

Answer 12

σ(u) = max(0,u).

Question 13

What is tanh activation?

Answer 13

σ(u) = tanh(u).

Question 14

What is the step activation?

Answer 14

σ(u) = 1_{u > 0}.

Question 15

What is the Swish activation?

Answer 15

σ(u) = u / (1 + e^(−βu)).

Question 16

What property do neural networks share with RBF kernels?

Answer 16

They are universal approximators: can approximate any continuous function on compact sets.

Question 17

Why use deep instead of shallow networks?

Answer 17

Deep networks approximate functions with fewer parameters than shallow networks.

Question 18

What is hierarchical feature learning?

Answer 18

Hidden layers learn progressively more abstract features (edges → shapes → object parts → concepts).

Question 19

Why does the last layer behave like a shallow model?

Answer 19

It is linear in its parameters: fθ(x)=ηᵀz where z is the final learned feature vector.

Question 20

How can we inspect what features a DNN learns?

Answer 20

By analysing which input patterns strongly activate specific neurons.

Question 21

What is data-driven feature learning?

Answer 21

Learning features automatically from data rather than manually designing them.

Question 22

Why are GPUs used for DNN training?

Answer 22

GPUs perform thousands of operations in parallel, enabling fast training of large models.

Question 23

What is a spurious feature?

Answer 23

A feature correlated with the label in training data but not truly relevant to the task.

Question 24

Example of spurious features?

Answer 24

Skin lesion classification using presence of a ruler as a signal for malignancy.

Question 25

What is an adversarial example?

Answer 25

Input with imperceptible noise that causes the DNN to misclassify.

Question 26

Why are adversarial examples dangerous?

Answer 26

They can fool autonomous vehicles, security systems, spam filters, etc.

Question 27

Why are adversarial features surprising?

Answer 27

They reveal the model uses patterns humans would never rely on.

Question 28

How does explicit regularisation work in DNNs?

Answer 28

Add penalties (e.g., L2) or use early stopping to avoid overfitting.

Question 29

What is early stopping?

Answer 29

Stop training when validation loss increases; acts as implicit regularisation.

Question 30

What is data augmentation?

Answer 30

Generate modified training samples (rotations, crops, synonym replacement, back-translation).

Question 31

Why is data augmentation useful?

Answer 31

Makes the model robust and reduces overfitting.

Question 32

What is dropout?

Answer 32

Randomly deactivate (e.g., 50%) neurons during training.

Question 33

Why does dropout help?

Answer 33

It prevents reliance on specific neurons and effectively trains many subnetworks.

Question 34

Are neurons dropped at test time?

Answer 34

No, all neurons are active at test time.

Question 35

How do we train DNNs?

Answer 35

Minimise empirical risk using mini-batch SGD or Adam with backpropagation.

Question 36

Why is backpropagation needed?

Answer 36

To compute gradients efficiently through the chain rule in deep composition functions.

Question 37

What is the chain rule for a 1-layer network?

Answer 37

∂l/∂w = (∂l/∂z)(∂z/∂a)(∂a/∂w) with a=wx+b, z=σ(a).

Question 38

Why is backprop efficient?

Answer 38

It reuses intermediate gradients instead of recomputing long chain-rule derivatives.

Question 39

What problem occurs without good initialisation?

Answer 39

Vanishing or exploding gradients.

Question 40

What is He/Kaiming initialisation?

Answer 40

Set W ∼ N(0, 2 / d_l) or N(0, 4/(d_l + d_{l+1})) to stabilise activations and gradients.

Question 41

Why does He initialisation work?

Answer 41

It keeps variances of activations and gradients consistent across layers.

Question 42

What optimiser is typically used?

Answer 42

Adam or mini-batch SGD.

Question 43

What are typical hyperparameters in DNNs?

Answer 43

Number of layers, width, activation, optimiser settings, learning rate, batch size.

Question 44

What are residual layers?

Answer 44

Layers of the form z_{l+1} = z_l + hθ(z_l).

Question 45

Why do residual layers help?

Answer 45

They smooth the loss surface and enable training of very deep networks (e.g., 1000+ layers).

Question 46

What trick helps residual networks avoid exploding gradients?

Answer 46

Batch Normalisation (instead of just He init).

Question 47

What is the purpose of a skip connection?

Answer 47

Allow gradients to flow easily across layers and improve training stability.