Feature Selection Algorithms

Question 1

Describe the 2 methods of feature selection

Answer 1

Scalar methods -
Consider each feature independently, evaluating its importance and relevance to the task. Do not take into account the relationships or dependencies between features

Vector methods -
Consider the joint distribution or relationships between features

Evaluate features in groups or as a whole, taking into account interactions between them

More computationally intensive than scalar, but can potentially lead to better feature selection when features interact in a complex way

Question 2

Key characteristics of scalar methods

Answer 2

Each feature is assessed on its own merit

They are simple and computationally efficient

Suitable for problems where features are independent or have minimal interactions

Question 3

Common techniques of scalar methods

Answer 3

Filter methods: Apply a statistical measure to evaluate the relationship between a feature and a target variable

Univariate selection / Statistical tests: features are ranked based on a statistical test and the top ranked ones are selected

Question 4

Key characteristics of vector methods

Answer 4

Features are considered together in their joint distributions

Can capture correlations and interactions between features

More computationally expensive but more powerful when dealing with correlated features

Question 5

Define wrapper methods

Answer 5

Vector methods which evaluate subsets of features by training a model on them and measuring the model’s performance

Question 6

3 examples of wrapper mtehods

Answer 6

Forwards selection: Starts with an empty set and adds the best features one by one

Backward elimination: Starts with all features and removes the least useful ones step by step

Recursive Feature Elimination: Recursively removes the least important features based on model performance

Question 7

Define embedded methods

Answer 7

Vector method that performs feature selection during the model training process

Question 8

Give 4 examples of embedded methods

Answer 8

Lasso:

Decision trees/Random forests

Principal Component Analysis

Independent Component analysis

Question 9

Describe the lasso method

Answer 9

Penalises the absolute value of coefficients, effectively setting some coefficients to zero

Question 10

Describe the decision tree/random forest method

Answer 10

Feature importance can be derived from tree-based models by examining the importance of each feature

Question 11

Describe the PCA method

Answer 11

Transforms the feature space into a new set of orthogonal axes, capturing the maximum variance of the data. Not strictly a feature selection method, reduces the feature space by creating a smaller set of uncorrelated features

Question 12

Describe the ICA method

Answer 12

Similar to PCA, ICA focuses on separating statistically independent components rather than uncorrelated components

Question 13

Describe the process of simple scalar feature selection

Answer 13

Choose a 1-dimensional class separability criteria, C. (Choose something that evaluates one feature at a time e.g. divergence)

The value of C(k) is computed for each feature, k

Select the n features corresponding to the n best values of C(k)

Simple to perform but does not consider correlation between features

Question 14

Describe the improved scalar feature selection method

Answer 14

Calculate the value of C(k) for each feature, k as before

Select the largest feature

Calculate C’(k) of the remaining features

C’(k) = C(k) - p(largest feature, current feature)

This will give the next best feature which doesn’t correlate with the feature already selected.

actual C’(k) has weights:
C’(k) = a1C(k) - a2p(largest feature, current feature)

Question 15

Describe the Sequential forward selection algorithm

Answer 15

Start with empty vector and progressively add features

At each iteration try adding each feature in turn to see which gives the best n-dimensional separability measure

Repeat until the vector is of the required length

Question 16

Describe the Sequential backward selection algorithm

Answer 16

Start by selecting all the features

Evaluate the impact of removing each feature one at a time

Select the feature whose removal has the least impact on the model performance and remove this

Keep selecting and removing until n features are left

Question 17

When to use forward/backward selection

Answer 17

Both are suboptimal

If the number of features is close to the desired number of features, choose backward selection

If the number of features is closer to 1, choose forward selection