Introduction

Question 1

Data

Answer 1

Found everywhere, generated at an unprecedented rate. Fundamental component of society

Question 2

Data in our lives

Answer 2

• Smart Home Devices
• Fitness Trackers (Track Biometric data)
• Financial Transactions

Question 3

Types of Data

Answer 3

Structured, Unstructured, Semi-Structured

Question 4

Structured Data

Answer 4

Well organized, formattable and easily searchable ex. Financial Records. Usually stored in RDBMS or files like csv

Question 5

Unstructured Data

Answer 5

Unorganized, Unformatted and different formats ex. Social Media Posts, Emails etc. Usually stored in file systems/CMS that preserve original structure

Question 6

Semi-Structured Data

Answer 6

Combination. Type of unorganized or partially organized data which doesn’t follow a rigid format but still has some level of structure. Mix of fixed and variable fields. Can be found in XML or JSON files.

Question 7

Qualitative Data examlpe

Answer 7

Gender, Nationality

Question 8

Quantitative Data example

Answer 8

Height, Weight

Question 9

Raw Data

Answer 9

Original Source of data, hard to use for analysis. Raw data may only need to be processed once

Question 10

Processed Data

Answer 10

Data that is ready for analysis, processing can include merging, subsetting, transforming etc. All steps should be recorded.

Question 11

Raw Data Example

Answer 11

ASCII files to Binary files that are machine generated, unformatted excel files, API responses.

Question 12

Accuracy

Answer 12

The measure of data quality that ensures data is correct, free from errors, and represents the real-world value accurately.

Question 13

Completeness

Answer 13

Indicates whether all required data is recorded or if some is missing/unavailable.

Question 14

Consistency

Answer 14

Ensures uniformity across data. Examples of issues include partially modified records or dangling updates.

Question 15

Timeliness

Answer 15

Refers to whether data is updated promptly to reflect the current state.

Question 16

Believability

Answer 16

Assesses how trustworthy or credible the data is.

Question 17

Interpretability

Answer 17

Reflects how easily the data can be understood by users.

Question 18

Data Consolidation Process

Answer 18

1. Moving data: Ensuring all data is gathered from different sources into a unified location.
2. Making it consistent: Aligning data formats and resolving inconsistencies.
3. Cleaning data: Removing errors, duplicates, and filling in missing values where possible.

Question 19

Why is Data Consolidation Needed?

Answer 19

• Data is often stored in different formats, making integration difficult.
• Data is frequently inconsistent across sources, causing discrepancies.
• Data may be dirty due to:
  
  • Internal inconsistencies (e.g., conflicting records).
  • Missing values or blank fields.
  • Potentially incorrect data caused by:
    
    • Faulty instruments.
    • Human or system errors.
    • Transmission errors during data movement.

Question 20

Disparate Data

Answer 20

Data is often stored in diverse locations and formats, which may include:
- Relational Databases: Used in operational systems for structured data.
- XML Files: Common in web services for hierarchical data.
- Desktop Databases: Such as Microsoft Access.
- Spreadsheets: Examples include Microsoft Excel.
- JSON: Popular for semi-structured or API-related data.

Question 21

Challenges with Disparate Data

Answer 21

• Data may reside on different operating systems
• Databases may operate on varying hardware platforms
• Integration of such varied data requires specialized tools and processes.

Question 22

Causes of Inconsistencies

Answer 22

• Faulty instruments, human/computer errors, transmission errors, or business requirements.
• Examples of discrepancies:
  
  • Two plants use different part numbers for the same item.
  • Systems using different formats for True/False (e.g., 1/0, T/F, Y/N).

Question 23

Other Data Quality Issues

Answer 23

• Free-form text entry:
  
  • Example: Same city entered as Louisville, Lewisville, and Luisville.
  • Cleaning routines must handle variations in bad data.
• Incomplete Data: Missing attribute values or only containing aggregate data.
  
  • Example: Occupation = “” or Gender = “Unknown”.
• Noisy Data: Contains noise, errors, or outliers.
  
  • Example: Salary = -10 (invalid).

Question 24

Causes of Missing Data

Answer 24

• Equipment malfunction.
• Deleted data due to inconsistencies.
• Misunderstandings or assumptions during data entry.
• Data not considered important at the time.
• Lack of historical records or change tracking.

Question 25

Handling Missing Data

Answer 25

1. Ignore the tuple: Common for classification tasks when class labels are missing. Not effective when missing values vary significantly. 2. Fill in missing values: - Manually: Labor-intensive and impractical for large datasets. - Automatically: - Global constant: e.g., `"unknown"`. - Attribute mean: Fill with mean value. - Class-specific attribute mean: Mean of samples from the same class. - Most probable value: Using inference (e.g., Bayesian formula, decision trees).

Question 26

Handling Noisy Data

Answer 26

1. Binning 2. Regression 3. Clustering 4. Hybrid Methods

Question 27

The Data Cleaning Process

Answer 27

1. Data Discrepancy Detecting 2. Data Migration and Integration

Question 28

Binning

Answer 28

- Sort data into bins, then smooth using: - Bin means. - Bin medians. - Bin boundaries.

Question 29

Regression

Answer 29

Fit data to regression functions for smoothing.

Question 30

Clustering

Answer 30

Detect and remove outliers as small clusters.

Question 31

Hybrid Methods

Answer 31

Combine automated detection with human inspection (e.g., flag suspicious values for manual review).

Question 32

Data Discrepancy Detection

Answer 32

- Use metadata (e.g., domain, range, dependency rules). - Verify field overloading, uniqueness rules, null values, etc. - Use tools for auditing: - Data scrubbing: Leverage domain knowledge (e.g., postal codes, spell-check). - Data auditing: Discover rules and relationships (e.g., correlations, clustering).

Question 33

Data Migration and Integration

Answer 33

- Data Migration Tools: Specify transformations for uniformity. - ETL Tools: Graphical interfaces to extract, transform, and load data. - Integrated Processes: Iterative and interactive approaches to improve accuracy and usability.

Question 34

Sharing Data

Answer 34

To effectively share data, ensure the following components are included: 1. Raw Data: The original dataset in its unaltered form. 2. Tidy Dataset: A processed version of the raw data with organized and meaningful structure. 3. Code Book: A document describing variables, values, and methods in the tidy dataset. 4. Step-by-Step Procedure: Detailed instructions to transform raw data into the tidy dataset and code book.

Question 35

The Code Book

Answer 35

essential to describe variables and provide context for the dataset.

Question 36

Sections of the Code Book

Answer 36

1. Variable Metadata: - Describe all variables in the raw dataset, including their units. - Document any summary choices made (e.g., averages, medians). 2. Study Design: - Thoroughly explain how the data was collected. - Include experimental study design details if applicable. - This section must be comprehensive and clear. 3. File Format: - Provide the code book as a standard document (e.g., `.txt`, `.docx`, `.pdf`).

Question 37

The Instruction List

Answer 37

A step-by-step guide to replicate the data cleaning and tidying process.

Question 38

Components of the Instruction List

Answer 38

1. Automated Scripts: Prefer scripts to automate the procedure for accuracy and reproducibility. 2. Inputs and Outputs: - Input: Raw data. - Output: Processed, tidy data. 3. Manual Steps: - Clearly explain any manual interventions. - Provide verification tasks and checklists to ensure correctness at each stage. 4. Intermediate Files: - Explain the purpose of any intermediate files generated during the process.

Question 39

The FASTA Format:

Answer 39

- The header line begins with `>` and contains: - gi: GenBank Identifier (e.g., `9626243`). - ref: NCBI Reference Sequence (e.g., `NC_001416.1`). - Description: Information about the sequence (e.g., "complete genome of Enterobacteria phage lambda virus").

Question 40

typical genome sequence consists

Answer 40

- A: Adenosine - C: Cytidine - G: Guanine - T: Thymidine - N: Any nucleotide (A, G, C, T).

Question 41

Transcribing the Code Book for Genome Data

Answer 41

Your Code Book should include: 1. Organism Name: The name of the organism (e.g., Enterobacteria phage lambda). 2. GenBank Identifier: If available, provide the GenBank ID (e.g., `9626243`). 3. NCBI Reference Sequence: Reference sequence, if available (e.g., `NC_001416.1`). 4. Subsequences: For subsequences, reference UCSC/Ensembl exon databases if applicable. 5. Study Variables: - Clearly list variables relevant to the study. - Example: For patient data, include attributes such as age, conditions (list), treatments (name, dose, unit).

Question 42

Study Design Section

Answer 42

Include a description of the technique used to collect data (e.g., sequencing methods).

Question 43

Instruction List for Data Preparation

Answer 43

- Include scripts or manual steps used for tidying the data. - Specify the file formats used to share data (e.g., `.csv`, `.fasta`). - Ensure reproducibility by providing detailed explanations of every processing step.

Question 44

Call Detail Records (CDRs) Key Observations

Answer 44

- Data lacks headers. - Missing data and gaps are present. - Entire records may be omitted if no activity exists for a combination of SquareID, Time Interval, and Country Code.

Question 45

Code Book Variables

Answer 45

- Square ID: ID of a square in the Milano GRID. (Type: numeric) - Time Interval: Start of time interval (milliseconds since Unix Epoch). End = Start + 600,000 ms (10 mins). (Type: numeric) - Country Code: Phone country code. Context varies by activity. (Type: numeric) - SMS-In Activity: Received SMS in the Square ID during the interval from the nation in `Country Code`. (Type: numeric) - SMS-Out Activity: Sent SMS in the Square ID during the interval to the nation in `Country Code`. (Type: numeric) - Call-In Activity: Received calls in the Square ID during the interval from `Country Code`. (Type: numeric) - Call-Out Activity: Issued calls in the Square ID during the interval to `Country Code`. (Type: numeric) - Internet Traffic Activity: Performed internet traffic in the Square ID during the interval by `Country Code`. (Type: numeric)

Question 46

Study Design

Answer 46

- File Format: TSV (Tab-Separated Values). - Missing Data: If no activity exists, the value is omitted. Empty fields indicate no record. - Dataset Origin: Aggregated from Telecom Italia cellular network’s CDRs in Milano for: - Received/Sent SMS. - Incoming/Outgoing Calls. - Internet activity (generated on start, end, or when 15 mins/5 MB limits reached). - Purpose: Measure mobile phone network activity by SMS, call, and internet usage. SMS and call data are comparable; internet traffic is not.