What is load balancing?
Ensuring each processing unit performs similar amount of work to minimize idle time
Why is load balancing important?
Poor balancing causes some processors to wait for slowest one, wasting resources
What is load imbalance overhead?
Time processors spend idle waiting for slowest processor to finish
What is the goal of load balancing?
Minimize idle time by equalizing work across all processors
What is static load balancing?
Distribute work at compile/startup time based on predicted workloads
What is dynamic load balancing?
Distribute work at runtime based on actual execution patterns
What is the advantage of static load balancing?
No runtime overhead, predictable
What is the disadvantage of static load balancing?
Can’t adapt to varying execution times
What is the advantage of dynamic load balancing?
Adapts to real execution patterns, handles heterogeneity
What is the disadvantage of dynamic load balancing?
Runtime overhead for task distribution
What is a task in load balancing?
Independent unit of work that can be executed by any processor
What is a work pool?
Centralized queue of tasks that processors request work from
What is the master-worker pattern?
One process distributes tasks, others execute and return results
What is MIMD?
Multiple Instruction Multiple Data - different processes execute different code
What is the difference between SIMD and MIMD?
SIMD: same instruction on different data; MIMD: different instructions on different data
What is the master process role?
Manages task queue, sends tasks to workers, collects results
What is the worker process role?
Requests tasks, executes them, returns results to master
What is the communication pattern in work pools?
Workers send requests to master, master sends tasks, workers return results
What is the scalability issue with centralized work pools?
Master becomes bottleneck with many workers
What are the solutions for scalability?
Hierarchical work pools, distributed work pools
What is the advantage of work pools?
Simple to implement, handles variable task sizes well
What is the disadvantage of work pools?
Communication overhead, potential master bottleneck
-
1-Introduction15
-
2-Multi-core-CPUs20
-
3-Data-Parallel-Problems19
-
4-Theory-of-Parallel-Performance18
-
5-Loop-Parallelism14
-
6-Critical-Regions-and-Atomics20
-
7-Locks-and-Synchronisation20
-
8-Distributed-Memory-Introduction18
-
9-Point-to-Point-Communication20
-
10-Parallel-Data-Reorganisation19
-
11-Reduction-in-Parallel19
-
12-Non-blocking-Communication21
-
13-Load-Balancing22
-
14-GPGPU-Introduction21
-
15-GPU-Threads-and-Kernels23
-
16-GPU-Memory-Types21
-
17-GPU-Synchronisation20
-
18-GPU-Atomic-Operations21
-
19-Task-Parallelism23
