Chapter 6.2 - Cross validation, bootstrap and tree-based methods - Tree-based methods

Question 1

What are tree-based methods?

Answer 1

Question 2

How do we write a classification problem of a regression tree in formal notation?

Answer 2

• A deep tree that has 100 leaves may only have one or two observations in each leaf –> thus do not generalise well
• A shallow tree that only has two leaves may not give us enough detailed insight into true classifications

Question 3

How do we go about finding c_j?

Answer 3

Target for regression:

• sum over all term regions, then look at all observations that fall into those regions, then calculate the L₂ loss for those observations

Question 4

How do we go about finding the partition order and deciding how the tree splits?

Answer 4

• why computationally intractable –> say we have a feature vector of length p. then if you want to find the optimal partition to minimise the the RSS then the computation cost is exp(p) –> which is going to become huge even when p is moderate
• Greedy: at the current time, look at what feature to add in or what split to take that has the best outcome currently (not the best outcome in the future or overall)
  • Risk of this is you can end up with a model that is very far away from being the optimal one

Question 5

How do we apply the top-down greed approach to build a tree model?

Answer 5

• In the bracket, we are basically calculating the minimal overall loss of all observations in the “left” region plus the “right” region. –> we set the estimators of c₁ and c₂ to be the average of the Y_i’s of the X’s belonging to that region
• There are only p variables to look at( so p possibilities for j) and at most n cut points (s). –> so computation cost is O(n *p) (CHECK THIS)

Question 6

What is the general process of splitting the predictor space?

Answer 6

1. You do the analysis to decide which feature you split and where
2. Then you can decide whether to start to split the left space or the right space next.

Question 7

How do we deal with the problem of overfitting when creating a tree through the top-down greedy approach?

Answer 7

Question 8

How do we go about choosing the optimal lambda (optimal subtree)?

Answer 8

Question 9

What is the entire algorithm for building a tree regression?

Answer 9

• Note: a tree with more splits will always have a lower training error (overfitting issue)

Question 10

How do tree methods change for classification problems?

Answer 10

l-hat(j) is the majority vote class that lies in the j-th region

• works well for a binary-classification problem

Question 11

What other methods of loss do we use for multi-classification problems?

Answer 11

• Gini measure purity or inequality – > pure class means that the leaf contains only all the observations of a single class –> will get a Gini coefficient of 0
• This implies the pure the class the lower the Gini, and the more uniform the class spread the higher the Gini values

Question 12

What are the advantages and disadvantages of the trees?

Answer 12

Question 13

How do we deal with the overfitting issue of trees?

Answer 13

Question 14

How do we perform Bootstrap aggregation or Bagging?

Answer 14

Question 15

Does bagging actually reduce variance?

Answer 15

• If there is correlation, we aren’t able to reduce variance as much if at all, when compared to the ideal independence sample case

Question 16

How can we improve on this for tree models?

Answer 16