Spark

Question 1

Driver Program

Answer 1

The process that runs the main function of an application and creates the SparkContext.

Question 2

Apache Spark

Answer 2

An open-source, distributed data-processing engine designed to handle real-time, batch, and iterative workloads efficiently.

Question 3

In-memory computing

Answer 3

A technique where data is cached in RAM to reduce repeated reads, leading to faster processing.

Question 4

saveAsTextFile

Answer 4

An action that writes the contents of a dataset to a text file.

Question 5

Spark Shell

Answer 5

An interactive environment for experimenting with Spark code.

Question 6

Tungsten Execution Engine

Answer 6

A project focused on improving the efficiency of memory and CPU utilization for computations.

Question 7

Cluster Manager

Answer 7

An external service in a Spark setup responsible for allocating resources to applications.

Question 8

Immutability

Answer 8

The property of an object whose state cannot be modified after it is created.

Question 9

MLlib

Answer 9

A library within for scalable machine learning algorithms.

Question 10

Broadcast Variables

Answer 10

Variables that are cached on each machine to avoid shipping a copy of large datasets with every task.

Question 11

flatMap

Answer 11

A data transformation that applies a function to each element of a data structure and flattens the results.

Question 12

Intersection Operation

Answer 12

Creating a dataset containing only the elements that are present in both of the input datasets.

Question 13

Map Operation

Answer 13

Applying a function to each element in a dataset and returning a new dataset with the transformed elements.

Question 14

MapReduce

Answer 14

A batch-oriented processing model known for its limitations in real-time, OLTP, graph, and iterative processing scenarios.

Question 15

Graph Processing

Answer 15

Analyzing relationships between entities represented as nodes and edges.

Question 16

RDD Lineage

Answer 16

The recorded sequence of operations that allows for the reconstruction of lost data partitions in a distributed dataset.

Question 17

Spark Packages

Answer 17

An ecosystem of extensions that add functionality to Spark.

Question 18

OLTP Workloads

Answer 18

Workloads characterized by short, numerous transactions.

Question 19

DataFrame

Answer 19

A distributed collection of data organized into named columns, offering a structured approach to data processing.

Question 20

Spark Streaming

Answer 20

A module designed for processing real-time data streams.

Question 21

Dataset

Answer 21

A typed version of a DataFrame, available in Scala and Java, that provides compile-time type safety.

Question 22

GraphX

Answer 22

A component in the ecosystem for graph-parallel computation.

Question 23

Shuffling

Answer 23

The process of redistributing data across partitions, often required for operations like joins and aggregations.

Question 24

Filter Operation

Answer 24

Selecting elements from a dataset based on a specified condition.

Question 25

Join Operation

Answer 25

Combining related information from multiple datasets based on a common key.

Question 26

WordCount

Answer 26

A classic example used to demonstrate distributed data processing, involving counting the occurrences of each word in a text.

Question 27

Data Replication

Answer 27

Creating multiple copies of data to ensure availability and fault tolerance.

Question 28

Resilient Distributed Dataset (RDD)

Answer 28

A fault-tolerant, immutable, distributed collection of objects that can be processed in parallel across a cluster.

Question 29

Lazy Evaluation

Answer 29

An evaluation strategy where expressions are not evaluated until their values are needed.

Question 30

Throughput

Answer 30

The amount of data that can be processed in a given unit of time.

Question 31

HDFS

Answer 31

A distributed file system designed for storing large datasets reliably.

Question 32

Accumulators

Answer 32

Variables that are only 'added' to through an associative and commutative operation and can be efficiently supported in parallel.

Question 33

High-level APIs

Answer 33

Programming interfaces that abstract away low-level details, making it easier to develop applications.

Question 34

Spark Core

Answer 34

The foundational module responsible for memory management, scheduling, fault recovery, and interaction with cluster managers.

Question 35

Executors

Answer 35

Processes that run computations and store data on worker nodes in a Spark cluster.

Question 36

Data Locality

Answer 36

The principle of moving computation close to the data to minimize network traffic.

Question 37

RDD Transformations

Answer 37

Operations on a dataset that are lazily evaluated and produce a new dataset, like map, filter, or union.

Question 38

SparkContext

Answer 38

The entry point to Spark functionality; it represents a connection to a Spark cluster and is used to create RDDs, accumulators, and broadcast variables.

Question 39

Fault Tolerance

Answer 39

The ability of a system to continue operating properly in the event of the failure of some of its components, achieved through mechanisms like lineage.

Question 40

Union Operation

Answer 40

Combining two datasets into a single dataset containing all elements from both.

Question 41

Spark SQL

Answer 41

A module for processing structured data, providing support for SQL queries and a DataFrame API.

Question 42

RDD Actions

Answer 42

Operations that trigger the execution of computations on a dataset and return results to the driver program, such as reduce, count, or first.

Question 43

Scalability

Answer 43

The ability of a system to handle increasing amounts of work or data without negatively impacting performance.

Question 44

Latency

Answer 44

The time it takes to process a unit of data.

Question 45

Batch Processing

Answer 45

Processing large volumes of data in a single job.

Question 46

Cartesian Product

Answer 46

Operation that combines each item of one RDD with each item of the second RDD

Question 47

Catalyst Optimizer

Answer 47

A query optimization framework within Spark SQL that improves query execution.

Question 48

Subtract Operation

Answer 48

Creating a dataset containing only the elements that are present in the first dataset but not in the second.

Question 49

Iterative Algorithms

Answer 49

Algorithms that repeatedly apply a set of operations to converge on a result.

Question 50

reduceByKey

Answer 50

A data transformation that merges the values for each key using an associative reduce function.

Question 51

Real-time Processing

Answer 51

Processing data as it is generated, providing immediate insights.

Question 52

Low-level APIs

Answer 52

Programming interfaces that provide fine-grained control over system resources and operations.

Question 53

RDD Partition

Answer 53

A logical division of data in a Resilient Distributed Dataset, enabling parallel processing.

Question 54

Worker Node

Answer 54

A machine in a Spark cluster that runs executors for computations.