Part 1

Question 1

Name a function that models the all-or-nothing response of biological neurons.

Answer 1

Threshold

Question 2

Why is the signum function not used in deep learning?

Answer 2

not differentiable at x=0
=> derivative of 0 or undefined → no learning would occur i.e. weights update would be 0 → never improve performance of network on the training data.

Question 3

What is Dropout?

Answer 3

• regularization technique
• prevent overfitting
• dropping out units (hidden + visible) in a NN

Question 4

What is the objective function of Rosenblatt’s perceptron?

Answer 4

Find weights that minimize the distance of misclassified samples to the decision boundary
Classification on the sign of the distance.

Question 5

Why is it useful to learn a bias term in training?

Answer 5

helps in offsetting the result,
reducing errors during computation by activation functions,
and ensuring a non-null output even when the input is null.
–> providing flexibility in shifting the activation function, thus improving the model’s ability to fit the data and make more precise predictions

Question 6

What is the task of Softmax function as the last layer in a neural network for a classification task?

Answer 6

produces a probability distribution over the classes for each input
By:
- rescale so that output sums up to 1
- produce non-negative output
= is normalized exponential function

Question 7

What are the advantages of making a neural network deeper?

Answer 7

• Greater expressive power
• model can scale to large and complex datasets
  Why?
  +more layers –> more diverse paths through which information can flow
  +exponential feature reuse + learns hierarchical representation of the data –> facilitates the extraction of increasingly abstract features

Question 8

What does backpropagation do?

Answer 8

computes all gradients required for the optimization of the network.

Question 9

What is the exploding gradient problem?

Answer 9

The updates in earlier layers can be increasingly large.
when the learning rate is too high –> positive feedback –> loss grows without bound

Question 10

What is the vanishing gradient problem?

Answer 10

The updates in earlier layers can be negligibly small.
When learning rate is too small –>negative feedback –> gradient vanishes

Question 11

What is the standard loss function for classification?

Answer 11

Cross-entropy loss: assumes that the outputs can be interpreted as probabilities that the input belongs to each class
Specifically, it assumes that the data follow a Bernoulli (for binary classification) or Multinoulli/Categorical (for multi-class classification) distribution.

Question 12

What is the standard loss function for regression?

Answer 12

L2-loss: assumes that the residuals (i.e. differences between the true and predicted values) follow a Gaussian distribution.

Question 13

What is Batch Gradient Descent? BGD

Answer 13

• Steepest GD
• computes the gradient of the cost function w.r.t the model parameters
• using the entire training dataset in each iteration

Question 14

What is Stochastic (Online) Gradient Descent?

Answer 14

• computes the gradient of the cost function w.r.t the model parameters
• use 1 sample in each iteration

Question 15

What is Mini-Batch SGD?

Answer 15

• computes the gradient of the cost function w.r.t the model parameters
• Use B «M random samples

Question 16

What are the steps to train a neural network using the backpropagation algorithm and an optimizer like Stochastic Gradient Descent (SGD)?

Answer 16

-randomly initiate the weights and biases
-forward the input into the network, get the output
compute the difference between the estimation and prediction
-tune the weights and biases of each neuron to minimize the loss
-iterate until the weights are optimized

Question 17

What is the idea of momentum-based learning?

Answer 17

Idea: Accelerate in directions with persistent gradients
-Parameter update based on current and past gradients i.e. use previous gradient directions to accelerate the training and become more robust against local minima.

Question 18

What is the purpose of the Momentum used in different optimizers?

Answer 18

It stabilizes the training by computing the moving average over the previous gradients.

Question 19

What is the zero-centering problem?

Answer 19

the lack of zero-centered output when the sigmoid function is used as an activation function in training neural networks.
–> covariate shift of successive layers

Question 20

how to solve the zero-centering problem?

Answer 20

Batch normalization which standardizes the inputs to each layer to have zero mean and unit variance, reducing the amount of internal covariate shift.

Question 21

What is the dying ReLUs problem?

Answer 21

• situation where a neuron gets stuck in the state where it only outputs 0 for any input
• A neuron’s weights get updated such that it starts to output 0 due to the input being negative → gradient for that neuron during backpropagation is also 0 →no more update

Question 22

What are the disadvantages of the powerful neural network with fully connected layers that motivate CNN?

Answer 22

• size problem: too many trainable weights –> too expensive
• pixels are a bad representation
  + Highly correlated neurons
  +Scale dependent –> struggle with images of different size
  + sensitive to Intensity variations
• doesn’t take into account the spatial relationships between pixels

Question 23

What are the advantages of CNN in comparison with fully connected neural networks?

Answer 23

-Local connectivity
-Weight sharing
-translational invariance (recognize patterns irrespective of their position in the input)
-grid-like alignment of images

Question 24

What decides the choice of function to apply to the output of CNN for the classification problems?

Answer 24

• Multi-class classification: each instance belongs to exactly 1 class. → probabilities of diff. classes to sum up to 1
  & make one of the probabilities significantly larger than the others → effectively deciding the class of the output. => use softmax function
• Binary classification: in multi-label classification, each instance can belong to more than one class → output independent probabilities for each class => sigmoid function

Question 25

What are the four essential building blocks of Convolutional Neural Networks?

Answer 25

- convolutional layer - activation function - pooling layer/ subsampling layer - Fully connected layer

Question 26

What is the function of the convolutional layer in CNN?

Answer 26

→ to detect local connectivity of features from the previous layers Sliding a set of filters (or kernels) across the input image. Each filter is responsible for learning some local feature within the image FEATURE LEARNING --> produce a feature map

Question 27

What is the function of the activation function in CNN?

Answer 27

provide nonlinearity to learn complex patterns

Question 28

What is the function of the pooling layer?

Answer 28

→ Compress and aggregate information across spatial location - save parameters, save computation, reduce overfitting - Downsampled feature maps

Question 29

What is the function of the fully connected layer in CNN?

Answer 29

-traditionally used at the end of CNNs for classification tasks. -connect every neuron to every neuron in the previous and subsequent layers, allowing the network to make predictions based on the high-level features learned in earlier layers.

Question 30

how can we do backpropagation through a convolutional layer?

Answer 30

convolving the output gradients with the flipped filter (horizontally and vertically)

Question 31

What is the purpose of convolution with 1x1 filter?

Answer 31

- Bottleneck - layer - to flatten or merge channels so that the size of the network decreases --> fewer parameters, save computations - It also reduces overfitting

Question 32

What are the benefits of 1x1 convolution?

Answer 32

1. 1x1 convolutions simply calculate inner products at each position 2. Simple and efficient method to decrease the size of a network 3. ** Learns dimensionality reduction, e.g., can reduce redundancy in your feature Maps

Question 33

how do we make CNN architecture better?

Answer 33

replace the last layer that fully connected with flatten + 1x1 or NxN convolution + global average pooling

Question 34

what is the bias-variance trade-off?

Answer 34

balance btw bias and variance in the performance of a ML model -bias: error -variance: model's sensitivity to fluctuations in the training data -Simultaneously optimizing bias and variance is impossible in general

Question 35

what does it mean by high/low bias/variance?

Answer 35

Low Bias, High Variance: -overly complex, capture noise - overfitting: well on training, poor on new unseen data High Bias, low variance: -too simple, underfitting -perform poorly on both training + test data Balance/sensible -capturing the underlying patterns without being too influenced by noise. -It generalizes well to new, unseen data.

Question 36

What is model capacity?

Answer 36

the capacity of a model describes a variety of functions it can approximate. related to nr. parameters

Question 37

How does the number of independent training samples affect the loss on the training and test set?

Answer 37

-Start with a small training data set --> suffer high test loss, low training loss, overfitting -Optimize variance by using more training data --> higher model capacity but match the size of the training set -More training data --> reduce test loss -Too high model capacity --> bad overfitting

Question 38

How does the model capacity affect the loss on the training and test set?

Answer 38

Increase model capacity → decrease training + test loss Up to overfitting point → start to produce bad overfitting: test loss increases --> Increase in bias for a decrease in variance

Question 39

Techniques to address overfitting in a NN

Answer 39

1. augment data 2. adapt architecture 3. adapt the training process 4. preprocessing 5. regularizer (in loss function) 6. dropout 7. use a validation set + use parameters with minimum validation loss as an early stop.

Question 40

How do we use data augmentation to avoid overfitting?

Answer 40

ensure: Every transformation to which the label should be invariant e.g. rotating transformation 1. random spatial transform 2. pixel transformation (change resolution, random noise, pixel distribution)

Question 41

What is the main idea of regularization in the loss function?

Answer 41

add a penalty term to the loss function

Question 42

What are ways for regularization in the loss function?

Answer 42

- enforce small norm: L2 norm - enforce sparsity: L1 norm

Question 43

How do the weights behave when the network is trained with L1 and L2 regularization? in comparison with a network without regularization

Answer 43

- L1 norm: +different shrinkage of weights +many weights are 0 esp. when lamda is large i.e. sparse - L2 norm: +weight decay (shrinkage) +small weight, more spread out or diffuse weight vectors

Question 44

What is the purpose of data normalization?

Answer 44

standardize the range of independent variables or features of data --> prevent domination

Question 45

How should data normalization be used in training NN?

Answer 45

- use training data ONLY - normalization of input data - normalization within the network

Question 46

What are some methods of data normalization?

Answer 46

-min/max -z-score / variance normalization - zero-centering / mean subtraction - Batch normalization

Question 47

What is the benefit of using Batch normalization?

Answer 47

- reduce Internal Covariate Shift i.e. It normalizes the distribution of the input for the layer that follows the Batch normalization layer - improve stability

Question 48

What is the internal covariate shift problem?

Answer 48

refers to the change in the distribution of network activations caused by adjustments in network parameters during training

Question 49

What are the reasons that leads to Internal Covariate Shift?

Answer 49

1. ReLU is not zero-centered 2. initialization and input distribution might not be normalized 3. deeper nets --> amplified effect

Question 50

What is a self-normalizing neural network?

Answer 50

- method to address the stability problem of SGD - SeLU + specific weight initialization - special form of dropout --> stable activations, stable training

Question 51

What is the aim of the dropout technique?

Answer 51

- reduce co-adaptation (different neurons in the network become highly dependent on each other during training --> less robust model) --> independent features

Question 52

What is the idea of dropout?

Answer 52

randomly drops/deactivates a fraction of neurons in the network at each update cycle i.e. ignored during the forward pass How: -set activations to 0 with (1-p) --> need compensation for dropout effect i.e. test time: multiply activation with p -drop connect (less efficient implementation)

Question 53

How would initialization affect convex optimization problems?

Answer 53

- does not matter i.e. always reach the global minimum - bad initialization--> slow convergence, more computational resources

Question 54

How would initialization affect non-convex optimization problems?

Answer 54

- does matter - NN with non-linearity is general non-convex

Question 55

How should biases be initialized?

Answer 55

- simply initialized to 0 - when using ReLU, a small positive constant e.g. 0.1 is better due to dying ReLU issue

Question 56

How should weights be initialized?

Answer 56

- random - bad to initialize with 0 - small uniform / Gaussian e.g. uniform random in the range [0,1]

Question 57

What is the idea of Xavier initialization?

Answer 57

-calibrate the variances for the forward pass by initializing with a zero-mean Gaussian -takes the number of input features into account

Question 58

What is the idea of He initialization?

Answer 58

effective when used with activation functions that have a mean close to zero e.g. ReLU - scale the weights by a factor that takes the number of input features into account --> helps keep the variance of the activations roughly the same across different layers.

Question 59

What is transfer learning?

Answer 59

reuse models / use a pre-trained model on a new problem - for a different task on the same data - on different data for the same task - on different data for a different task

Question 60

How does transfer learning work?

Answer 60

- weight transfer (diff task) e.g. pre-trained model --> image classification --> target model: object detection/ segmentation - transfer between modalities (diff data type)

Question 61

What are the benefits of transfer learning?

Answer 61

- weight transfer: + Capitalizes on features learned by the source model, + speeding up training on the target task. - transfer between modalities +benefit from the representation learning achieved in one modality when training on a different modality. +Useful when labeled data is scarce in the target modality.

Question 62

What is multi-task learning (MTL)?

Answer 62

train a network simultaneously on multiple related tasks

Question 63

What is hard parameter sharing in MTL?

Answer 63

- several hidden layers are shared between all tasks (usually feature extraction layer) - MTL of N tasks --> reduce chance of overfitting by an order of N

Question 64

What is soft parameter sharing?

Answer 64

Each model has its parameters Instead of forcing equality, the distance between parameters is regularized as part of the loss function Options e.g. l2-norm, trace-norm, ...

Question 65

What are auxiliary tasks for?

Answer 65

- to create a more stable network - additional tasks to the original task e.g. facial landmark detection + learning subtly related tasks e.g. face pose, smile/not smile, glasses/no glasses, gender