Backprop

Question 1

Finite difference rule and its difficulty

Answer 1

f’(x)= (f(x+h) - f(x))/h, difficulty = numerical instability for small values of h

Question 2

Taylor Expansion

Question 3

Saddle points/increasing/decreasing functions

Question 4

Gradient Descent Update

Answer 4

theta_t <– theta_{t-1} - lr * grad of f at theta_{t-1}