Data Processing

Question 1

What is batch processing?

Answer 1

Batch processing handles large volumes of data at scheduled intervals. Data is collected, processed in groups, and output produced. Used for analytics, ETL, reports. Examples: Hadoop MapReduce, Spark batch jobs.

Question 2

What is stream processing?

Answer 2

Stream processing handles data in real-time as it arrives. Continuous processing with low latency. Used for real-time analytics, monitoring, fraud detection. Examples: Apache Kafka Streams, Flink, Storm.

Question 3

What is MapReduce?

Answer 3

MapReduce is a programming model for processing large datasets. Map phase transforms data into key-value pairs. Reduce phase aggregates values by key. Enables parallel distributed processing.

Question 4

What is ETL (Extract Transform Load)?

Answer 4

ETL extracts data from sources, transforms it (clean, aggregate, format), and loads into target system (data warehouse). Used for data integration and analytics pipelines.

Question 5

What is a data warehouse?

Answer 5

A data warehouse is a centralized repository for structured data from multiple sources. Optimized for analytics and reporting with historical data. Examples: Snowflake, BigQuery, Redshift.

Question 6

What is a data lake?

Answer 6

A data lake stores raw, unstructured data in native format until needed. Schema-on-read approach. More flexible than data warehouse but requires processing. Examples: AWS S3 + analytics tools.

Question 7

What is data partitioning in analytics?

Answer 7

Partitioning divides large datasets into smaller chunks based on criteria (date, region). Improves query performance by scanning only relevant partitions. Common in big data systems.

Question 8

What is OLTP vs OLAP?

Answer 8

OLTP (Online Transaction Processing): handles day-to-day transactions, fast writes, normalized. OLAP (Online Analytical Processing): handles complex queries, read-heavy, denormalized, historical data.

Question 9

What is a materialized view?

Answer 9

A materialized view is a precomputed query result stored as a table. Trades storage and update cost for faster reads. Must be refreshed when source data changes.

Question 10

What is data sharding in analytics?

Answer 10

Sharding distributes data across nodes for parallel processing. Each shard contains subset of data. Enables horizontal scaling for large datasets. Sharding key determines data distribution.