Chapter 19

Question 1

What are ETL pieces

Answer 1

Transformation etc

Question 2

What is data cleansing

Answer 2

Dirty data should remove to go to dataware house

Question 3

What is GIGO

Answer 3

Stands for “Garbage In, Garbage Out.” GIGO is a computer science acronym that implies bad input will result in bad output.

Question 4

What is dirty data

Answer 4

It is relative term. It means data does not confirm its value.

Question 5

Who tells data dirty or clean

Answer 5

The person who have domain knowledge

Question 6

What is toddler employee

Answer 6

Example of dirty data. Employee too much young to get a job

Question 7

What is un-born employee

Answer 7

Employee DOB is less than Date of joining

Question 8

What is govt decision making

Answer 8

Investment of govt where there is no need and it is loss of money

Question 9

What is direct mall marketing

Answer 9

Failure of advertisement campaign and loss of money

Question 10

What are lighter side of dirty data

Answer 10

• Toddler Employee

- Un-born Employee

Question 11

What are 3 classes of anomalies

Answer 11

• Syntactically dirty data
• Semantically dirty data
• Coverage anomalies

Question 12

What are sub classes of syntactically dirty data

Answer 12

• Lexical errors

- Irregularities

Question 13

What are sub classes of Semantically dirty data

Answer 13

• Integrity constraint violation
• Business rule contradiction
• Duplication

Question 14

What are Coverage anomalies

Answer 14

• Missing attributes

- Missing Records

Question 15

What are lexical errors

Answer 15

There is problem in structure of data and storage problem

Question 16

What are irregularities

Answer 16

Missing of unit (e.g. there is salary in column 2000 and we do not know it is Pkr, USD or what)

Question 17

What is Integrity constraint violation

Answer 17

Integrity constraint violations occur when an insert, update, or delete statement violates a primary key, foreign key, check, or unique constraint or a unique index.

Question 18

What is business rule contradiction

Answer 18

It is violation of business rule

Question 19

How we can handle coverage anomalies

Answer 19

• Remove that record that have problem
• Manual data feeding
• Use global constant figure (use global value and use it where ever missing)
• Replace most probable value with missing value

Question 20

What are 2 key based problem

Answer 20

• Primary key problems

- None-Primary key problems

Question 21

What are primary key problems

Answer 21

• Same key but different data
• Same entity with different keys
• PK in one system but not in other
• Same PK but not in different formats

Question 22

What are non primary key problems

Answer 22

• Different encoding in different sources (e.g. M/F and some place male/female and so on)
• Multiple ways to represent the same information
• Sources might contain invalid data
• Two fields with different data but same name
• Required field left empty
• Data incomplete
• Data contain null values

Question 23

What are 4 methods of automate data cleansing

Answer 23

1- Association rules (Make rules on statistical properties)
2- Pattern based (Find different pattern values)
3- Statistical (with the help of mean value etc)
4- Clustering (group together values which are similar and anomalies left alone)