How do you clean and preprocess data in python? (AI)

Question 1

Front (Question/Concept)

Answer 1

Back (Answer/Explanation)

Question 2

Which Python library is used for the majority of data manipulation tasks, including loading, filtering, and cleaning?

Answer 2

pandas (DataFrames)

Question 3

What is the primary library used for numerical operations (like array handling)?

Answer 3

numpy

Question 4

What is the pandas code to load a CSV file named data.csv?

Answer 4

data = pd.read_csv('data.csv')

Question 5

What is the simplest pandas method to remove records (rows) with missing values?

Answer 5

.dropna(inplace=True)

Question 6

What is the pandas method to replace missing values with a calculated statistic like the mean?

Answer 6

.fillna(data.mean(), inplace=True) (This is a form of Imputation.)

Question 7

What is the goal of Standardization (scaling)?

Answer 7

Transforms data to have a mean of 0 and a standard deviation of 1.

Question 8

Which sklearn class performs Standardization?

Answer 8

StandardScaler

Question 9

What is the goal of Normalization (scaling)?

Answer 9

Scales data to a specific range (usually 0 to 1).

Question 10

Which sklearn class performs Normalization?

Answer 10

MinMaxScaler

Question 11

What technique groups numerical data into categories (e.g., ‘Low’, ‘High’)?

Answer 11

Data Binning/Categorization

Question 12

What is the pandas function used for Data Binning?

Answer 12

pd.cut()

Question 13

What technique converts categorical data to numerical format by creating binary columns for each category?

Answer 13

One-Hot Encoding

Question 14

What is the pandas function used for One-Hot Encoding?

Answer 14

pd.get_dummies(data, columns=['...'])

Question 15

What technique converts categorical data by assigning an integer to each category?

Answer 15

Label Encoding

Question 16

Which sklearn class performs Label Encoding?

Answer 16

LabelEncoder

Question 17

Why might Label Encoding be problematic for non-ordinal categorical features?

Answer 17

The model may mistakenly assume a hierarchical order or magnitude relationship between the assigned integers.

Question 18

What is the goal of Feature Engineering?

Answer 18

To create new features (e.g., data['new_feature'] = data['f1'] * data['f2']) that provide better insight for the model.

Question 19

What is the sklearn function used to split data into training and testing sets?

Answer 19

train_test_split (from sklearn.model_selection).

Question 20

What common statistical method is used for Outlier Detection?

Answer 20

Z-scores or Interquartile Range (IQR) methods.

Question 21

What is the pandas method used to remove duplicated rows?

Answer 21

.drop_duplicates(inplace=True)