Module 1 - Core data concepts

Question 1

Data classified (3)

Answer 1

Structured
Semi-Structured
Unstructured

Question 2

Structured Data

Answer 2

• Has a predifined schemea every record has the same fields and properties.
• Tabular format:
• rows are instances
• columns are attributes

Question 3

Semi-structured data

Answer 3

• Has some structure, but allows flexibility and variation acrosss different instances.
• Common formats:
• JSON
• MXL
• CSV files

Question 4

Unstructured data

Answer 4

Lacks a predifined model or consistent organizational structure.
- Ei.:
- Text documents, images, videos…

Question 5

Catagories of data stores (2)

Answer 5

• File stores
• Databases

Question 6

File Storage explained

Answer 6

• Can be local or central (cloud)
• File formats:
• Type of data being stored
• The application or services that will access it
• Whether it needs to be readable to humans or be optimized for storage.

Question 7

Delimited Text Files (File types)

Answer 7

• Plain text, good for structured data
• CSV
• commas separate the fields
• rows are terminated be a carriage return or a new line
• Others:
• TSV, Space-delimited, Fixed-width data

Question 8

JSON (File types)

Answer 8

• Layered format, with objects that have multiple attributes
• Objects are enclosed in curly braces
• Collections or arrays are enclosed in square brackets
• Attributes are expressed as name-value pairs, separated by commas

Question 9

XML (File types)

Answer 9

• Uses tags (in angle-brackets) and indents to show elements and attributes

Question 10

Binary Large Object (BLOB) (File types)

Answer 10

• For unstructured data
• Data stored in raw binary values, that needs to be interpreted
• Images, videos, audio …

Question 11

(Storage) Optimized File Formats

Answer 11

• Designed for efficient storage and processing.
• Can be compressed, indexed and efficiently stored and retrieved.
• Types: Avro, ORC, Parquet

Question 12

Avro (Optimized File Formates)

Answer 12

• Avro records are made up of two parts
• Header is in Json, defines the structure of data in the record.
• Data itself in in binary
• Good for compression and reduced storage or network bandwidth usage.

Question 13

ORC (Optimized File Formates)

Answer 13

• Columnar layout, devided into stripes
• One stripe contains data for a specific column or group of columns.
• One stripe has:
• Index (Access to records in stripe)
• Actual data
• Footer (Statistical data for column)

Question 14

Parquet (Optimized File Formates)

Answer 14

• Columnar layout
• Supports compact storage, high-performance read and write operations, and works well with complex data types.
• Organized into row groups

Question 15

Databases explained

Answer 15

Centralized and specialized system to store data, and enable querying.

Question 16

Relational Databases

Answer 16

• Designed to store and query structured data.
• Organized in tables, instances are given a primary key to be able to refer to them

Question 17

Non-Relational Databases (NoSQL Databases)

Answer 17

They do not enforce a realational schema on the data they store.

Question 18

Key-value (NoSQL Databases categories)

Answer 18

-One record is made up of
- a unique key
- an associated value that can be of any format

Question 19

Document (NoSQL Databases categories)

Answer 19

-One record in made up of
- a unique key
- a JSON document

Question 20

Column Family (NoSQL Databases categories)

Answer 20

• Tabular structure
• One Column can have more Columns inside it, called column families. They are related columns.

Question 21

Graph (NoSQL Databases categories)

Answer 21

Data is represented as nodes, that are entities with links (edges) used to define the relationships between them.

Question 22

Transactional data processing

Answer 22

• Systems that capture transactions, events that need to be monitored
• Commonly this work in called Online Transactional Processing (OLTP)

Question 23

OLTP systems (Transactional data processing)

Answer 23

• Depend on optimal read and write performance
• Support for standard CRUD operations
• Require transactions to follow ACID properties

Question 24

ACID properties (Transactional data processing)

Answer 24

• Atomicity
• Consistency
• Isolation
• Durability

Question 25

Atomicity (ACID properties)

Answer 25

Ensure that transactions are indivisible unit of work. Be completed in full or fail entirely.

Question 26

Consistency (ACID properties)

Answer 26

Transactions can only move the database from one valid state to another.

Question 27

Isolation (ACID properties)

Answer 27

Transactions running at the same time do not interfere with one another.

Question 28

Durability (ACID properties)

Answer 28

Guarantee that if a transaction is committed, it is permanent and will remain committed.

Question 29

Analytical data processing

Answer 29

Relies on read-only systems, that store large amounts of historical information or business metrics

Question 30

The most common architecture for enterprise-scale analytics

Answer 30

1 - Operational data is extracted, tranformed, and the loaded (ETL) into a datalake to be analyzed 2 - Data is then loaded into a set of tables. It might be a lakehouse or data warehouse. 3 - Data is then further aggregated and loaded into an OLAP model 4 - Now the data stored can be queried to generate reports, dashboards and visualizations

Question 31

Data warehouse explained

Answer 31

- For storing data in a relational schema - Read-heavy workloads (queries) Structured, relational storage optimized for fast analytical queries and reporting.

Question 32

Data lake explained

Answer 32

- Large-scale analytical processing scenarios where file-based data needs to be gathered and analyzed. Stores large valumes of raw data (structured and unstructured) as files.

Question 33

Data lakehouse explained

Answer 33

- New architecture, the combination of the lake and warehouse. - Scalable, flexible -> lake - Relational querying features -> warehouse