Chapter 5.2 - Classification (Part II) - Kernel Methods

Question 1

What is SVM, and how does it relate to SVC?

Answer 1

H

Question 2

How do we use basis expansion in SVC optimisations to enlarge the space of features by including squared terms?

Answer 2

Question 3

If we use a basis expansion of cubic polynomials on (x⁽¹⁾, x^{(2)</sup), how does the SVC optimisation problem change?}

Answer 3

Question 4

What is the problem with specifying a large basis?

What method do we use to efficiently enlarge our feature space to accommodate a nonlinear boundary between classes?

What is the key idea about SVC we will be using?

Answer 4

Question 5

How can both the MMC and the SVC be represented as their inner product?

Answer 5

• Complexity of inner product = O(p) –> cost saving over standard basis expansion

Question 6

What is the informal idea behind using the kernel and its association with the inner product?

Answer 6

Question 7

How do we conduct SVM using a kernel?

Answer 7

• Benefit of the kernel method: Dont need to know what high-dimensional feature we are working in, or even need to compute the inner products, if we have a function k(x,x’) that denotes the relationship of the inner products when transformed into the higher-dimensional spaces.

Question 8

Can we take the original ridge regression optimisation problem and represent it in the form of an inner product?

Thus, can we apply the kernel method under the ridge regression?

Answer 8

• Feb 19th (2 ) March –> 37mins –> WORK through algebra
• Matrix XX^T every element becomes the pairwise inner product

Question 9

What is the summary of the intuition behind using the kernel?

So what question do we have when using the kernel?

Answer 9

Question 10

What is the mathematical setup, features and variables used in the kernel method?

(6 bullet points)

Answer 10

Question 11

How do we define the (real) vector space?

What is an inner product space? What properties does it have to satisfy?

What is the notion of “length” given by in this space?

What is a Hilbert space?

Answer 11

Question 12

What is the formal definition of a Kernel?

Answer 12

Question 13

What is the formal definition of a (Positive semi-definite kernel)?

Answer 13

Question 14

What is the definition of a positive semi-definite matrix?

Answer 14

kernel <–> positive-semi definite kernel

Question 15

Given k₁ and k₂ are kernels, what else is a kernel?

Answer 15

• third is the pointwise limit
• if k_i = constant –> this is always a kernel

Question 16

What is the:

• Linear kernel
• polynomial kernel
• Gaussian kernel

Answer 16

Question 17

What are the three equivalent definitions of a Kernel: Moore-Aronszajn Theorem?

Answer 17

Question 18

How do we build towards the reproducing kernel Hilbert space ( RKHS)?

Answer 18

• We can construct the Hilbert spaces from any linear combinations of the φ_j
• z = predictor

Question 19

What is the formal definition of a Reproducing Kernel Hilbert Space (RKHS)?

Answer 19

• The reproducing kernel of any RKHS is a kernel
• For any positive semi-definite kernel k, there exists a (unique) associated RKHS with k being its reproducing kernel (Moore–Aronszajn theorem).

Question 20

What is the Representer Theorem?

Answer 20

• You are trying to find F that minimises the empirical risk + penalty term with the penalty being a strictly increase funciton of the norm of f squared. –> where you are working in a RKHS

Question 21

With a kernel, what does the SVM classifier for a new observation depend on?

Answer 21

Question 22

What are four popular kernels?

Answer 22

Question 23

How do SVM work with a radial kernel?

Answer 23

Question 24

What is the advantage of kernel methods?

Answer 24