lesson 7

Question 1

how to Verify Data Quality? 3

Answer 1

DOCUMENTATION

– CONTENT:
- Entity (what is about?)
- Property
- Measurement (units) type of variable
- Time

– TECHNICAL
- Abbreviations, codes
- Program code for data set creation, conversion

TRUTHFULNESS
- verify from other sources
-plausibility

COMPLETENESS

– TALL:
- missing observations?
– WIDE:
- encoding for missing documentation (NaN, 0..)

Question 2

how to preserve data quality?

Answer 2

CONVERSION:

– DATA TRANSFORMATION
- convert unit
-aggregation
MERGING:
- ‘key’ is key

– DATA CLEANING
- limit the scope of the analysis (focus on the scope!)
- check the realistic/possible range of value
- check the origin of the data
- eliminate outliners
- eliminate border observation

Question 3

how can we treat the missing data?

Answer 3

ELIMINATE: horizontally or vertically but cannot be a statement

IMPUTE: make a statement based on other variables. (estimation)

INTERPOLATION: male a statement based on the same variable. OK CROSS-SECTION, NOT OK Interpolation in time

IMPROVE ECONOMETRICS: other methods - different form interpolation

Question 4

what is meant by Audit trail?

Answer 4

document the entire process from original data to the final result

Question 5

6 dimension of data quality

Answer 5

1. Accuracy -> Data is accurate when it reflects reality. es. height of a person is recorded as 15 cm
2. Completeness->Data is complete when all required data for a particular use is present.
3. Uniqueness->Data is unique if each entry appears only once within a dataset, without duplicates
4. Consistency-> Consistency is achieved when data values do not conflict within a record or across
5. Timeliness-> Data is timely if it is available when expected and needed.
6. Validity->Data is valid if it conforms to the expected format, type, and range. es. @ nell’email.