Activation Functions

Question 1

Softmax

Answer 1

• activation function
• common in multi-class classification
• transforms a vector of real numbers into a probability distribution, where each element represents the probability of the input belonging to a specific class.

Question 2

ReLU

Answer 2

• activation function
• Rectified Linear Unit
• Outputs input directly if it’s positive. Outputs zero otherwise.
• ReLU(x) = max(0, x)

Question 3

What are the benefits of ReLU?

Answer 3

• Adds non-linearity to the network enabling it to learn complex patterns in the data
• Computationally inexpensive
• Alleviates vanishing gradient problem that can occur with sigmoid and tanh fns during backpropagation in deep networks

Question 4

Sigmoid

Answer 4

• activation function
• transforms input into an output between 0 and 1 (a probability)
• often used in binary classification

Question 5

tanh

Answer 5

• hyperbolic tangent
• activation function
• squashing function
• maps a wide range of values to -1 to 1

Question 6

what’s faster to train with sigmoid or tanh?

Answer 6

• tanh because it is zero centered
• tanh has steeper gradient around the center compared to sigmoid which can mitigate the vanishing gradient problem during backpropagation

Question 7

Vanishing gradients

Answer 7

occurs when gradients, used to update neural network weights during training, become extremely small as they propagate backward through the network