id5059

Question 1

What is the purpose of boosting methods in ensemble techniques?

Answer 1

To train a sequence of simple predictors that correct the errors of previous ones

Boosting methods improve the performance of weak classifiers.

Question 2

Name the two boosting methods discussed.

Answer 2

• AdaBoost (Adaptive Boosting)
• Gradient Boosting

These methods are used to enhance the performance of weak classifiers.

Question 3

In AdaBoost, what happens to the training data instances when training the ith predictor?

Answer 3

Instances are weighted to increase the influence of those misclassified by the ensemble

This helps improve the accuracy of the ensemble model.

Question 4

What is the AdaBoost model based on?

Answer 4

A weighted voting classifier

The predictions are made based on the performance of the predictors.

Question 5

In AdaBoost, what does a weight of 2 for a training instance imply?

Answer 5

The instance contributes twice as much to the objective function

This means it has a greater influence on the training process.

Question 6

What is the initial weight of each training instance in AdaBoost?

Answer 6

w(i)1 = 1/m for all i

Here, m is the number of training instances.

Question 7

What is a key characteristic of Gradient Boosting?

Answer 7

Each predictor is trained to directly correct the errors of the previous ones

This sequential training improves the overall model accuracy.

Question 8

In Gradient Boosting for Regression, what is the initial value of F0(x)?

Answer 8

F0(x) = 0

This serves as the starting point for the predictions.

Question 9

What does the learning rate parameter (η) do in Gradient Boosting?

Answer 9

Controls the contribution of each predictor to the final prediction

Affects how quickly the model learns.

Question 10

True or false: In AdaBoost, the estimators can be trained in parallel.

Answer 10

FALSE

Estimators in AdaBoost must be trained sequentially.

Question 11

What is a common base classifier used in AdaBoost?

Answer 11

Small decision trees

However, any classifier can be used.

Question 12

What is a drawback of ensemble methods like AdaBoost and Gradient Boosting?

Answer 12

Loss of interpretability

The complexity of the model makes it harder to understand.

Question 13

What is the final ensemble in Gradient Boosting based on?

Answer 13

The sum of the predictions of the base predictors

This aggregation improves the overall prediction accuracy.

Question 14

What is the role of validation in boosting methods?

Answer 14

To determine the learning rate and the number of predictors to use

Helps in tuning the model for better performance.

Question 15

What is the main goal of ensemble learning?

Answer 15

To combine the predictions of many weak predictors to make a strong predictor

This enhances the overall predictive performance.

Question 16

What is the curse of dimensionality?

Answer 16

The phenomenon where algorithms and ML models see degradation in performance on high-dimensional data

This can affect computational cost and model performance.

Question 17

List two issues caused by the curse of dimensionality.

Answer 17

• Computational cost
• Model performance

High-dimensional space has counter-intuitive geometry leading to these issues.

Question 18

What does dimensionality reduction aim to achieve?

Answer 18

Creating a new representation of a data set in fewer dimensions while preserving its structural properties

This process can help mitigate issues related to high-dimensional data.

Question 19

Name two benefits of dimensionality reduction.

Answer 19

• May reduce overfitting
• Speeds up model training

Some models perform better in lower dimensions.

Question 20

What is a major drawback of dimensionality reduction?

Answer 20

Loses information

Ground truth decision boundaries may become more complex.

Question 21

What are projection techniques in the context of dimensionality reduction?

Answer 21

A class of linear dimensionality reduction techniques

Examples include principle component analysis (PCA) and random projection.

Question 22

What does Principle Component Analysis (PCA) find?

Answer 22

The dimensions of maximum variation in the data

Finding principle components is equivalent to finding a singular value decomposition (SVD).

Question 23

What is the equation for the PCA decomposition?

Answer 23

X = UΣV⊺

Where U, V, and Σ are matrices representing the decomposition.

Question 24

How can you reduce dimensions using PCA?

Answer 24

By projecting to the first k principle components: Tk = XVk

Vk contains the first k columns of V.

Question 25

What does the **squared entries of Σ** represent in PCA?

Answer 25

Proportional to the variance of the data explained by each principle component ## Footnote This helps in understanding the importance of each component.

Question 26

What is **Incremental PCA**?

Answer 26

An algorithm for approximately computing the PCA decomposition by processing the dataset in batches ## Footnote Useful for large datasets.

Question 27

What is the **Johnson-Lindenstrauss Lemma (JL Lemma)**?

Answer 27

States that if k ≥ 4 log(n)(1/2 ϵ² - 1/3 ϵ³), then distances between points are preserved ## Footnote This lemma supports the effectiveness of random projections.

Question 28

List two **advantages** of Random Projection.

Answer 28

* Simple to implement * Fast to compute ## Footnote Effective on very high-dimensional data.

Question 29

What is a **disadvantage** of Random Projection?

Answer 29

Requires more dimensions than PCA ## Footnote Dense random projection has a high memory overhead.

Question 30

What is a **manifold** in the context of topology?

Answer 30

A topological space that locally resembles Euclidean space near each point ## Footnote In data science, a manifold is a low-dimensional structure embedded in a higher-dimensional space.

Question 31

What happens to the **manifold structure** of data when using projection techniques?

Answer 31

It can be lost ## Footnote This is a concern in manifold learning where the goal is to discover the structure of the manifold.

Question 32

What is the goal of **manifold learning**?

Answer 32

To discover the structure of the manifold ## Footnote This involves not simply projecting the data along some axis but may involve stretching or twisting the data.

Question 33

What is **Local Linear Embedding**?

Answer 33

A manifold learning method that finds a low dimensional embedding of the data preserving relative positions of nearby points ## Footnote It represents each point as a linear combination of its k nearest neighbors.

Question 34

List the steps involved in **Local Linear Embedding**.

Answer 34

* Represent each point as a linear combination of other data points * Restrict to k nearest neighbors * Minimize reconstruction error * Normalize the weights * Find low dimensional points minimizing reconstruction error * Use weights calculated from the original dataset ## Footnote It preserves local distances well but not far distances and has a high computational cost.

Question 35

What are some **other manifold learning methods**?

Answer 35

* Multidimensional Scaling * Isomap * t-SNE ## Footnote Each method has its own approach and computational considerations.

Question 36

What does **Multidimensional Scaling** aim to achieve?

Answer 36

Find a low dimensional embedding preserving distances between all pairs of points ## Footnote It is computationally expensive with a complexity of O(n^3).

Question 37

How does **Isomap** work?

Answer 37

Builds the nearest neighbor graph and embeds it to preserve geodesic distances ## Footnote It is effective and popular but expensive in high dimensions.

Question 38

What is the purpose of **t-Distributed Stochastic Neighbor Embedding (t-SNE)**?

Answer 38

A visualization method for high dimensional data that keeps similar instances close and dissimilar instances apart ## Footnote The algorithm follows a complex optimization process, which can be computationally expensive.