Principal Component Analysis

Question 1

Curse of Dimensionality

Answer 1

• Standard regression classification techniques can become :
• ill-defined for M >> N
• ill conditioned/ numerically unstable even for M < N
• increase in dimensionality > exponential increase of space > data becomes sparse
• amount of data neede for a reliable result often grows exponentially with the dimensionality

Question 2

Regularization

Answer 2

• Idea: impose constraints on the parameters to stabilize solution
• example: introduce prior probability

Question 3

Maximum a-posteriori approach

Answer 3

Question 4

Dimensionality Reduction

Answer 4

• Goal: reduce data to features most relecant for learning task
• i.e. significance test for single features; find relevant directions/subspaces in correlated data

WHY?

• Vizualisation
• better generalization
• speeding up
• data compression

Question 5

Principle Component Analysis

Answer 5

• assume data is centered
• find direction of maximum variance
• Eigenvaluze Problem, direction of largest variance corresponds to direction of largest eigenvector
• not robust to outliers

Question 6

PCA applications

Answer 6

• Dimensionality reduction
• Eigenfaces
• Denoising

Question 7

Power Iteration

Answer 7

• Why: full eigendecomposition of scatter matrix is slow, often interesten only in a few first principal components
• Power Iteration Method: start with random vector w, parameter update w<- Sw/||Sw||