7: Machine Learning 2

Question 1

What are two models for supervised learning?

Answer 1

• KNN classification
• Decision trees

Question 2

What is the K-nearest neighbours (KNN) algorithm?

Answer 2

A supervised machine learning algorithm that classifies a new data point into the target class, depending on the features of its neighbouring data points.

• Assumptions: the new observation is likely from the same group as its closest neighbours, the data point that have the most similar characteristics.
• Measure distance with the Euclidian distance

Question 3

What are the steps in using the KNN algorithm?

Answer 3

1. Choose the number K of nearest neighbours (rule of thumb: K = sqrt(n), where n is the sample size).
2. For any unlabelled training point (observation, know values but not label), identify the K nearest labelled points (shortest distance).
3. Assign to the unlabelled point the most frequent label of the K nearest labelled point.

Question 4

What is Kappa in the KNN output?

Answer 4

Kappa = (Accuracy - random accuracy) / (1 - random accuracy)

A negative value would mean that random assignment would do better than our prediction.

Question 5

What are decision trees used for?

Answer 5

Trees are a way to split data into purer subsets. We try to find the easiest way to group data in the most homogeneous categories, iteratively.

Question 6

What does the process of decision trees look like?

Answer 6

1. Find the best split into the data: the split leading to the purer subgroups.
2. Split the data into branches.
3. Do it again at the next level.

• Start with root node (containing all training data).
• Aims to stop at leaf nodes (final decision, pure or almost pure classes)
• Pure class = only contains data from the same group
• internal nodes = intermediary nodes

Question 7

How do we find the best split for decision trees?

Answer 7

The Gini impurity.

1. Try all possible splits in our data at this level.
2. For each possible split, calculate the average level of impurity (gini).
3. Best split = lowest impurity. Always try to minimise Gini impurity.

Question 8

What is a limitation of the Gini impurity?

Answer 8

It is computed only for the possible splits of one node, so it can only reach a local minimum (algorithm might not identify the optimal tree overall).

Question 9

What are some limitations of decision trees classifiers?

Answer 9

1. The tree is only optimal locally.
2. Small changes in the training dataset might lead to large changes in the classification rules.
   –> Decision trees are unstable!