Dimensionality Reduction

Question 1

What is Dimensionality Reduction?

Answer 1

It is the process of reducing the number of variables under consideration by obtaining a smaller set of principal variables.

Question 2

Methods to implement dimensionality reduction?

Answer 2

It can be implemented in two ways:
1. Feature Selection
2. Feature Extraction

Question 3

What is Feature Selection?

Answer 3

Here we are interested in finding k of the total n features that give us the most information, and we discard the other (n-k) dimensions.

eg: subset selection method(forward and backward selection)

Question 4

What is Feature Extraction?

Answer 4

Here, we are interested in finding a new set of k features that are combinations of original features.

Question 5

How the error is measured in machine learning problems?

Answer 5

In regression:
We usually use the Mean Squared Error(MSE) or the Root Mean Squared Error(RMSE).

In Classification:
Here, we may use the misclassification rate as the measure of error.

Question 6

What is MSE?

Answer 6

It is the sum of the square of the difference between the predicted and the actual target variables, divided by the number of data points.

Question 7

What is the misclassification rate?

Answer 7

It is the ratio of misclassified examples by the total number of examples.

Question 8

Why is dimensionality reduction useful?

Answer 8

1. Decreasing the size of the dataset also decreases the complexity of the inference algorithm during testing.
2. Saving the cost of extracting unnecessary data.
3. Simpler models have less variance.
4. We get a better idea about the process that underlies the data, which allows knowledge extraction.
5. data with lesser dimensions can be plotted and analyzed visually for structure and outliers.

Question 9

What is Subset Selection?

Answer 9

It is also known as feature selection or variable selection or attribute selection. It is the process of selecting a subset of relevant features for use in model selection.

Question 10

What are the two approaches in subset selection

Answer 10

1. Forward Selection
2. Backward Selection

Question 11

What is Forward selection? Explain in detail.

Answer 11

Here, we start with no variables and add them one by one such that the variables that decrease the error at the most get added. The addition occurs until there is no more decrease in the error rate.

Explain the algorithm.

Question 12

What is Backward Selection? Explain in detail.

Answer 12

Here, we start with the set containing all the variables(features) and at each step, we remove that variable that causes the least error.

Explain the algorithm.

Question 13

What is a Principal Component Analysis (PCA)?

Answer 13

It is a statistical procedure thet uses orthogonal transformation convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components.

The number of principal components is always less than or equal to the number of original number of variables/observations.

Question 14

What are the various steps in PCA? Explain.

Answer 14

1. Consider a dataset having n features and N examples.
2. Compute the means of the variables.
3. Calculate the covariance matrix (S).
4. Calculate the eigen values and eigen vectors of the covariance matrix(S).
5. Derive a new dataset.
6. New Dataset.
7. Conclusion.