Data_Quality (Brohrer)

Question 1

choose your rows

Answer 1

one and only one target value for each row (well… for multilabel, multiclass may not be true but still it has to answer a precise question as in single label case)

Question 2

inspect the data

Answer 2

• spot problems in data
• identify the domain of values for different features, use functionality similar to DataFrame.describe() - pandas
• review the documentation for all features
• visualize data (histograms, scatter plots, etc) for anomalies, correlation, etc.

Question 3

correct the data

Answer 3

• if the error is obvious, correct the data

- otherwise delete the value and leave it as missing

Question 4

missing data -> mean

Answer 4

• assumes the missing data is distributed in the same manner as the present one
• aka Missing Completely at Random

Question 5

missing data -> median, mode

Answer 5

• median is a robust statistic useful when dealing with outliers
• mode is used when dealing with categorical variable.s

Question 6

missing data -> interpolated value

Answer 6

• interpolate the value when dealing with ordered data

- useful with time-series data

Question 7

missing data -> constant

Answer 7

• when dealing with information that is not Missing at Random i.e. people don’t share income info as it is very ‘high’; using high gives a better representation of missing info even if not precise

Question 8

missing data -> missing rank

Answer 8

• when dealing with data presented in ranking order with no duplicates or gaps

Question 9

missing data -> 0

Answer 9

• when dealing with numerical data is a reasonable choice

- when dealing with categorical data use it as (key, value) pair (0, ‘missing’)

Question 10

missing data -> 0 - confidence

Answer 10

• if confindence in using 0 is low, use a separate column/feature to indicate with a boolen flag the missing value

Question 11

missing data -> ‘missing’

Answer 11

• use such an indicator value or similar for categorical data

Question 12

missing data -> delete columns

Answer 12

• use it when the number of missing data instances is to high to be useful (uninformative)
  !!! NOTE:
• make sure there is enough data left before proceeding further

Question 13

missing data -> delete rows

Answer 13

• use it when critical information (features) per row is missing
  !!! NOTE:
• make sure there is enough data left before proceeding further

Question 14

missing data -> imputation

Answer 14

• a way to have information from other columns used to fill missing information in a reasonable way (i.e. through some correlated values between features)