VA Session 6 !

Question 1

Variable types

Answer 1

• Numerical = numbers, ordered set & magnitude known
• Continuous = infinite e.g. temperature, age
• Discrete = finite e.g. shoe size, nr of children
• Categorical = words (also discrete)
• Nominal = no order / hierarchy e.g. gender (M,F) e.g. hair color
• Ordinal = order / hierarchy but magnitude btw successive values not known e.g. clothing size (XS, S, M)

Question 2

Formats to store data

Answer 2

• Wide = each column represents a different variable
• Long = one column contains all the variables & another gives the value for that variable

Question 3

Exploratory Data Analysis

Answer 3

• Summary statistics
• visually exploring (important, e.g. of dragon data where different visualisations with same summary statistics)

Question 4

Summary Statistics

Answer 4

• Quick overview & formulate new hypotheses
• Calculating statistics: mean, median, mode, variance, standard deviation, range
• Check datatypes

Question 5

Matplotlib

Answer 5

• “standard” plotting library for Python
• Pro: visualizations are extremely customizable
• Con: syntaxy little complicated (many lines of code)

Question 6

Seaborn

Answer 6

• higher-level library based on Matplotlib (<-> prettier plots with fewer lines)
• Works with Pandas DataFrames, arrays & NumPy arrays & standard Python structures (e.g. lists & dictionaries)
• can access columns of DataFrame just by column name

Question 7

Scatter plots

Answer 7

• for inspecting relationship btw two variables (bivariate data), reveal e.g. clusters or correlations between variables
• e.g. Ice Cream Consumption & temperature of day

Question 8

Histograms

Answer 8

• understand distribution (frequency) of variables
• continuous (not finit) variable & quantiative data (frequency)
• con: bin size (how often data divided) -> can significantly affect interpretation

Question 9

Density Plots

Answer 9

• understand distribution of variables
• continuous (not finit) variable & density
• • not issue of bin size
• • difficult to interpret

Question 10

Line Charts

Answer 10

• good to visualize time series data, easy to see trends
• If “jumps”: look closer, could be a special occasion or measurement mistake)
• plot 95% confidence interval & mean

Question 11

Bar Chart

Answer 11

-discrete variable (countable, finit) & categorical data
- e.g. countries & body_mass_index

Question 12

Boxplot

Answer 12

• comparing distributions in categorical data
• outliers
• max, min
• middle = 50% of data points (27th & 75th percentile)
• middle = medin

Question 13

Heatmaps

Answer 13

• see relationship between 2 variables & their different values
• color = magnitude of some measurement
• e.g. number of flights by month & year, smaller numbers = darker

Question 14

Cleaning & preprocessing data - Common issues

Answer 14

• Missing values
• Faulty data
• Wrong data type
• Duplicates (found during exploration)

Question 15

Data Cleaning

Answer 15

• Structuring data
• identifying & handling missing values (Remove rows, columns)
• Drop unnecessary variables (columns)
• checking for outliers
• Convert fields with text -> numeric
• Scaling or normalizing data
• Creating new variables
• Renaming variables (remove capitalization, spaces)

Question 16

Removing values vs imputing

Answer 16

Depends on situation

Question 17

Removing values

Answer 17

• Less risky
• But: could make dataset too small or imbalanced
• Remove columns with lots of missing values rather than rows

Question 18

Imputing

Answer 18

• replacing missing data with substituted values
• use non-missing values of same feature (column) to give substitute, e.g. mean, mode, median
• Univariate imputation: use one column
• Multivariate imputation: use (entire) set of features (columns)
• nearest neighbor imputation: replaced by a value obtained from related cases in the whole set of record

Question 19

Problem & Methods for scale of data

Answer 19

• some statistical methods & machine learning algorithms = sensitive to scale or assume normality e.g. one variable range of 0-1, other 100-10.000 (plotting could look very strange)
• Normalization
• standardization
• log transformed

Question 20

Creating new variables

Answer 20

sometimes good to combine features into one feature by creating a ration of 2
e.g. Combining weight & height to BMI

Question 21

Normalization

Answer 21

• same scale, from 0 to 1
• sklearn.preprocessing.MinMaxScaler()

Question 22

Standardization

Answer 22

• data points expressed as Standard derivation from mean
• sklearn.preprocessing.StandardScaler()