L12 - Dimensionality Reduction

Question 1

What is meant by dimensionality reduction?

Answer 1

The reduction of feature count within a data set.

Question 2

What 3 ways does dimensionality reduction improve / enhance the creation and running of ML models?

Answer 2

Saves time, saves money, removed irrelevant data.

Question 3

What are the 2 methods of dimensionality reduction? Define each…

Answer 3

1. Feature extraction -> Extract useful combinations of features from the data.
2. Feature selection -> Analyse all features of the data to establish the relevant ones.

Question 4

What are the 3 methods for feature selection?

Answer 4

1. Filter Methods
2. Wrapper Methods
3. Embedded Methods

Question 5

Explain the Filter Method…

Answer 5

• Method of feature selection for dimensionality reduction
  1 -> Bring features to the same scale through normalisation or standardisation
  2 -> Choose some variance threshold
  3 -> Calculate the variance of each feature, dropping ones that are below the threshold

Question 6

Explain the Wrapper Method…

Answer 6

1. Method of feature selection for dimensionality reduction
2. Can either be conducted via Forward Search or Recursive Feature Elimination
3. Both FS and RFE conduct a battle royale process to establish best features.
4. Both stop when there are N models each with N best features

Question 7

Explain how Forward Search works in Wrapper Method of Feature Selection…

Answer 7

1. Create N models with 1 feature each
2. Find the best feature E.g Feature 3
3. Create N-1 models each with previous best feature (F3) + another feature E.g Model1(F3,F1), Model2(F3,F2) etc…
4. Repeat until we have a models with N features and can choose the best one

Question 8

Explain how Recursive Feature Elimination works in Wrapper Method of Feature Selection…

Answer 8

1. Reverse of Forward Search
2. Start with N-1 models each with N-1 features
3. Repeatedly remove the single worst feature from each model
4. Result in final best model with M features

Question 9

Explain the Embedded Method

Answer 9

1. Method of feature selection for dimensionality reduction
2. Use decision trees to establish the best features
3. The use random forests to aggregate the result of the decision trees

Question 10

What is a Random Forest?

Answer 10

An aggregation of decision trees

Question 11

What are the 2 types of methods for Feature Extraction?

Answer 11

1. Linear
2. Non-Linear

Question 12

What is the main Linear method for feature extraction? Explain it…

Answer 12

1. Principal Component Analysis
2. Find an orthogonal coordinate transformation such that every new coordinate is maximally effective
3. This creates new N variables, named Principal Components
4. Principal Components are linear combinations of the original coordinates
5. The orthogonal coordinate with the most variation is the most informative

Question 13

What is the worst case scenario of PCA?

Answer 13

1. When all variables are equally important, but are uncorrelated.
2. This provides us with no information.

Question 14

What are the steps of PCA?

Answer 14

1. Generate the covariance matrix from the dataset
2. Diagonalise the covariance matrix
3. Multiply XV
4. Take first K principal components with the largest eigenvalues.
5. This gives us a K-dimensional representation of the data, having extracted the K most important features.
6. The dimensionality reduction comes from the removal of least important principal components

Question 15

What do the Eigenvalues represent in PCA?

Answer 15

The variance capture by each principal component.

Question 16

What is the Input-Output into PCA’s?

Answer 16

Input -> High-D data
Output -> Low-D data

Question 17

What are the 2 main Non-linear methods for feature extraction?

Answer 17

1. t-SNE
2. UMAP

Question 18

Explain t-SNE

Answer 18

A non-linear method for feature extraction

1 - Calculate the distribution of distances across the N points and call this D

2 - Scatter N points randomly in 2 or 3 dimensions

3 - Move the N points around until distance distribution resembles D

Question 19

What are some issues with t-SNE?

Answer 19

• Faraway points are meaning less
• Poor scaling due to high memory usage

Question 20

Explain UMAP

Answer 20

1. A non-linear method for feature extraction
2. Runs faster and uses less memory than t-SNE

Question 21

What are some issues with both t-SNE and UMAP?

Answer 21

1. Hyperparameter dependence
2. Cluster sizes and distance between clusters mean nothing
3. X and Y axis are almost impossible to interpret