Session 4.1

Question 1

Bias-Variance tradeoff

When trying the optimal model we are in fact trying to find…

Answer 1

the optimal tradeoff between bias and variance

Question 2

Bias-Variance tradeoff

We can reduce variance by

Answer 2

by putting many models together and aggregating their outcomes

Question 3

Bagging (or bootstrap aggregation) creates

Answer 3

multiple data sets from the original training data by bootstrapping – re-sample with repetition.

Runs several models and aggregates output with a voting system

Question 4

Other ensemble methods

Random Forest

Answer 4

combines bagging with random selection of features (or predictors)

Question 5

Other ensemble methods

Boosting

Answer 5

applies classifiers sequentially, assigning higher weights to observations that have been mis-classified by the previous methods

Question 6

A table model

Answer 6

memorizes the training data and performs no generalization

Useless in practice! Previously unseen customers would all end up with
“0% likelihood of churning”

Question 7

Generalization

Answer 7

is the property of a model or modeling process whereby
the model applies to data that were not used to build the model

If models do not generalize at all, they fit perfectly to the training data !
–> they overfit

Question 8

Overfitting

Answer 8

is the tendency to tailor models to the training data, at the expense of generalization to previously unseen data points.

Question 9

Holdout Validation

Answer 9

1. Given only one data set, we split it into a a training set used for fitting the model and a test set used for evaluating the model
2. Performance is evaluated based on accuracy in the test data a.k.a. “holdout accuracy”
3. Holdout accuracy is an estimate of “generalization accuracy”

Question 10

As a model gets more complex, it is allowed to pick up harmful spurious correlations

Answer 10

• These correlations do not represent characteristics of the population in general
• They may become harmful when they produce incorrect generalizations in the model

This phenomenon is not particular to decision trees

• It is also not because of atypical training data
• There is no general analytic way to avoid overfitting

Question 11

Simplest method to limit tree size:

Answer 11

specify a minimum number of instances that must be present in a leaf

Question 12

Just as with trees, as you increase the dimensionality,

Answer 12

you can perfectly fit larger and larger sets of arbitrary points

• Often, modelers manually prune the attributes in order to avoid overfitting
• There are ways to select attributes automatically

Question 13

Why is overfitting bad?

Answer 13

A small imbalance in the training data can be ’learned’ by the tree and erroneously propagated

Question 14

Why is the phenomenon of overfitting not particular to decision trees

Answer 14

• It is also not because of atypical training data

- There is no general analytic way to avoid overfitting