7. Tree-based algorithms

Question 1

Purpose of tree-based algorithms

Answer 1

Prediction

( applicable to regression and classification )

Question 2

Impurity measures

Answer 2

Used to determine the quality of the split
- Gini Index
- Entropy
- Re-substitution error

Question 3

Explain decision tree alg

Answer 3

(0. Pre-processing e.g. binarization, else use regression tree instead of classification tree)
1. Recursive binary splitting and determine splits via impurity measures
2. Improve with Cost complexity pruning –> Grow large tree and prune it back.
( Select tuning parameter using (k-fold) cross-validation )

Question 4

Decision Tree advantages

Answer 4

1. Low pre-processing effort (normalization, scaling not required)
2. Missing values have little effect
3. Easy to explain and interpret (closely mimic human decision-making)
4. Handle qualitative predictors without need to create dummy variables ( qualitative, quantitative, continuous, and discrete variables )
5. Faster than RF

Question 5

Decision Tree disadvantages

Answer 5

1. Lower predictive accuracy (vs. Regression / Classification)
   –> improve by aggregation, at loss of interpretability and speed
2. Overfitting risk (unlike Bagging/RF)
3. Instability for changes in data
4. Can become quite Complex (expensive)

Question 6

Effect of Bagging, Boosting, RF

Answer 6

+ Increase predictive accuracy (lower variance)

• Lower interpretability
• Lower speed (complexity)

RF: Adds random selection to bagging, to produce uncorrelated trees. Less risk of overfitting

Question 7

Explaining Bagging

Answer 7

• Many weak learners (bootstrap samples) are trained.
• Take the mean of these estimation over the collection of bootstrap samples
  –> The overall prediction is the most commonly occurring class among the predictions (== majority vote)

Question 8

Explain Boosting

Answer 8

System of ensemble learners, using a gradient boosting function to iteratively train models that use data values that have been modelled poorly in previous iterations.

( Bags chosen at random with replacement = bootstrap samples)
+ Improve performance and reduce variance

Question 9

Boosting vs. RF vs. Bagging

Answer 9

Boosting:
— Selects from all predictive variables
— Sequentially depends on error rate of previous iteration
— 3 Tuning Param
Benefit: Learns with previous error term

RF:
— Selects from subset of predictive variables
— Built independently at each iteration
— 2 Tuning Param

Bagging:
— Aggregation (by mean) of bootstrap samples