Session 4

Question 1

A fitting graph shows

Answer 1

the accuracy (or error rate) of a model as a function of model complexity

Question 2

Fitting graph

Generally, there will be more overfitting as…

Answer 2

one allows the model to be more complex

Question 3

Complexity is a measure of

Answer 3

the flexibility of a model

Question 4

If the model is a mathematical function, complexity is measured by

Answer 4

the number of parameters

Question 5

If the model is a tree, complexity is measured by

Answer 5

the number of nodes

Question 6

To look at overfitting, you only look at the

Answer 6

holdout data

Question 7

Generally: A procedure that grows trees until the leaves are pure tends to overfit

Answer 7

• If allowed to grow without bound, decision trees can fit any data to arbitrary precision
• The complexity of a tree lies in the number of nodes

Question 8

Ways to avoid overfitting for tree induction

Answer 8

• Stop growing the tree before it gets too complex

- Prune back a tree that is too large (reduce its size)

Question 9

Tuning model parameters

When choices all are made, then test on test set this “nested” test set is often called the

Answer 9

“validation” set (to differentiate from the final test set).

Question 10

Cross-validation

Answer 10

the dataset was first shuffled, then divided into ten partitions

Question 11

How to find which variables are the most important for the model?

Answer 11

There are multiple ways of determining how important is a variable:

• (Weighted) Sum of information gain in each split a variable is used (tree-based models)
• Difference in model performance with and without using that variable (all models)

Question 12

“Random Forest”

Answer 12

is a tree-based model that uses multiple decision trees simultaneously

A “Random Forest” model is equivalent to a decision tree if:

• We set the parameter “number of trees” to 1; and
• We set the parameter “subset ratio” to 1

Rapidminer has an operator called “Weight by Tree Importance” that calculates the variable importance based on information gain (among others)
This operator only works with models of type “Random Forest”

Question 13

A learning curve is…

Answer 13

a plot of the generalization performance (testing data) against the amount of training data

Question 14

Learning curve

Generalization performance improves as…

Answer 14

more training data are available

Steep initially, but then marginal advantage of more data decreases

Question 15

The ROC graph shows

Answer 15

the entire space of performance possibilities for a given model, independent of class balance

Question 16

A ROC curve can be constructed from

Answer 16

selecting different thresholds from a rank classifier

Question 17

The area under the ROC curve

Answer 17

is the probability that the model will rank a randomly chosen positive case higher than a negative case

Question 18

AUC is useful when

Answer 18

a single number is needed to summarize performance, or when nothing is known about the operating conditions

Question 19

Comparing models

We can compare the performance of different models by looking

Answer 19

at their ROC curves.

We can choose the optimal model for each threshold

Question 20

The expected value framework is …

Answer 20

an analytical tool that is extremely helpful in organizing thinking about data-analytic problems.

Question 21

The expected value framework combines:

Answer 21

1. Structure of the problem
2. Elements of the analysis that can be extracted from the data
3. Elements of the analysis that need to be acquired from other sources (e.g., business knowledge)

Question 22

The benefit/cost matrix

Answer 22

summarizes the benefits and costs of each potential outcome, always comparing with a base scenario