8-Distributed-Memory-Introduction

Question 1

What is distributed memory parallelism?

Answer 1

Multiple processes with separate memory spaces that communicate explicitly over a network

Question 2

Why is explicit communication needed?

Answer 2

Processes cannot directly access each other’s memory - must send/receive messages

Question 3

What is the key difference from shared memory?

Answer 3

No data races possible, but communication overhead is the main performance bottleneck

Question 4

What is a process vs a thread?

Answer 4

Process: separate address space, expensive to create; Thread: shared address space, cheap to create

Question 5

How do processes communicate?

Answer 5

Via interprocess communication (IPC) like sockets, pipes, or message passing libraries

Question 6

What is MPI?

Answer 6

Message Passing Interface - standard library for distributed memory programming

Question 7

What are the advantages of distributed systems?

Answer 7

Scalability to thousands of nodes, no cache coherency issues, fault tolerance

Question 8

What are the disadvantages?

Answer 8

Communication overhead, explicit data movement, more complex programming

Question 9

What is a cluster?

Answer 9

Collection of computers connected by high-speed network, running same OS

Question 10

What is a supercomputer?

Answer 10

Large-scale cluster optimized for high performance computing

Question 11

What is the fastest supercomputer today?

Answer 11

El Capitan: 11,340,000 cores, ~1.8 exaFLOPS, 30MW power consumption

Question 12

What are exaflop machines?

Answer 12

Supercomputers capable of 10^18 floating point operations per second

Question 13

What is the interconnection network?

Answer 13

High-performance network connecting cluster nodes (e.g. InfiniBand, Gigabit Ethernet)

Question 14

What is the difference between local and wide area networks?

Answer 14

Local: high bandwidth, low latency; Wide area: lower bandwidth, higher latency

Question 15

What is the primary performance bottleneck in distributed systems?

Answer 15

Communication time - data transfer between nodes

Question 16

What makes distributed programming challenging?

Answer 16

Coordination without shared clock, explicit data movement, potential network failures

Question 17

What is the benefit of distributed systems for large problems?

Answer 17

Can solve problems too big for single machines

Question 18

What types of applications benefit most from distributed computing?

Answer 18

Weather forecasting, scientific simulations, large-scale data processing