Dimensionality Reduction by Linear Transformation

Question 1

What is the curse of dimensionality

Answer 1

A phenomenon in which data analysis and ML becomes difficult in high-dimensional spaces

Data becomes more difficult to visualise - hard to plot greater than 3d spaces

Computational cost explodes as more variables to process

Data becomes sparse

Number of samples require grows exponentially

Models struggle to generalise

More difficult to find patterns

Distance between samples becomes pointless because it is so big

Question 2

Problem with feature selection

Answer 2

Most feature selection methods evaluate the importance of the feature independently, this means when considering a pixel in an image as a feature, the value of the pixel is only meaningful in the context of it’s neighbours

An edge, a curve or a line is not a single pixel, it’s a pattern of adjacent, highly correlated pixels

Question 3

Describe the process of dimensionality reduction by linear transformation

Answer 3

x = {x1,x1,x3, … ,xN} is the vector of all features

We define a new vector y = {y1,y2,y3, …, yM}

where M < N

We consider a class of functions, H, known as linear transforms

y1 = a11x1 + a12x2 + … + a1NxN
y2 = a21x2 + a22x2 + … + a2NxN
yM = aM1x1 + aM2x2 + … + aMNxN

These equations can be written more compactly as:
y = Ax where A is the M x N matrix of parameters aij

Then, for each y, we set one a to 1 and all the rest to 0

Example: y1 = 1x1 + 0x2

Image shows an example

Question 4

What is flattening

Answer 4

Converting from 2d to 1d e.g. converting a 28x28 matrix to a single 1x784 vector