! Session 2: Pre-Processing & EDA

Question 1

Cleaning - examples

Answer 1

o Convert fields with text into numeric form
o Remove rows with missing / faulty data
o Fix rows with faulty datapoints
o Drop unnecessary variables (columns)
o Scaling or normalizing data
o Creating new variables
o Renaming variables (remove capitalization, spaces)

Question 2

Methods to changing categorical (& ordinal) to numerical values

Answer 2

• one hot encoding
• label encoding
• ordinal encoding

Question 3

one hot encoding

Answer 3

= transforming each possible value from categorical into new binary (dummy-variable) column (e.g. true = 1, false = 0)

Question 4

label encoding

Answer 4

= transforming each possible value from categorical into a unique integer value
! Issue: Algorithms might wrongly misjudge order / relationship in data

Question 5

Ordinal encoding

Answer 5

= transforming each possible value from categorical into a unique integer value where values maintain order (e.g. body mass g in size)

Question 6

EDA

Answer 6

• Exploratory Data Analysis
• approach for data analysis that employs a variety of techniques (mostly graphical) to
• uncover underlying structure
• extract important variables
• detect outliers and anomalies
• test underlying assumptions

Question 7

2-D Histogram

Answer 7

= dividing points among 2D bins

Question 8

Skewness

Answer 8

= measure of asymmetry of probability distribution of variable about its mean

Question 9

Boxplot

Answer 9

= method for graphically depicting groups of numerical data through their quartiles

Question 10

Scaling & Normalizing - Relevance

Answer 10

• Some learning algorithms are sensitive to the scale differences in variables
• e.g. Distance bases Algorithms: knn & svm
• e.g. 1€ !=1dkk

Question 11

Mathematical Transformation

Answer 11

• scaling & normalizing
• log, exp
• standardization

Question 12

Standardization

Answer 12

data points expressed as SD from mean
Make mean 0 and variance 1

Question 13

Scaling

Answer 13

• adjusts the range of feature values to a specific range
• e.g. between 0 and 1

Question 14

Pipelines

Answer 14

• Data transformation steps needed to be executed multiple times in right order
• Method: Pipeline class from Scikit-Learn
• Pipeline constructor takes a list of name/estimator pairs defining a sequence of steps

Question 15

Splitting of data

Answer 15

• Training set: train the model
• Test set: confirm that the model works
• Validation: tune the hyperparameters -> Optional

Question 16

Validation set - purpose

Answer 16

• ML = iterative process; fine tuning
• Used to ensure model can generalize, used during hyperparameter tuning
• Test set: just for final evaluation

Question 17

Normalizing

Answer 17

• transform features to have a common scale by adjusting the distribution of data (standardization a form of normalizing)
• centers the data around zero & adjusts the spread of values