Perceptron Learning Algorithm P2

Question 1

How do you find the best set of weights (W’)

Answer 1

Define the cost function J(W’)
Apply the gradient descent algorithm (iterative algorithm that finds the minimum of the cost function)

Question 2

Define the cost function J(W’)

Answer 2

It is the sum of losses L(W’) for the misclassified samples
If the sample is misclassified as positive and is actually negative, it is multiplied by -1 and then added to the sum

Question 3

Is J(W’) always positive or negative

Answer 3

Positive. as L(W’) is always positive because if the sample being added is negative, it is *-1

Question 4

Define L(W’)

Answer 4

This function measures the distance between a misclassified sample and the decision boundary

The first variable on the RHS is either 1 or -1 depending on the sign of the misclassified sample

Question 5

Describe gradient descent to minimise the cost function

Answer 5

The gradient of the cost function dJ/dw1 shows the slope of J at any point w1

If the gradient is positive we move left, towards the negative

If the gradient is negative, we move right towards the positive

We are trying to find the minimum of the cost function

Question 6

What is the formula for the new set of weights using the gradient descent rule

Answer 6

= W’(t) - rate of learning * gradient of the cost function

The gradient of the cost function is a vector of partial derivatives that points in the direction of the steepest increase in cost

Question 7

What is the main effect of choosing a too large learning rate in gradient descent

Answer 7

The updates may overshoot the minimum and bounce back and forth

Question 8

Advantage and disadvantage of having a small learning rate

Answer 8

It is more stable but converges more slowly

Question 9

Formula for the new set of weights using gradient dexcent

Answer 9

Old weights - learning rate * derivative of loss function