17-GPU-Synchronisation

Question 1

What is the scalar product?

Answer 1

a · b = Σ(a_i × b_i) for vectors a and b

Question 2

What is the serial scalar product algorithm?

Answer 2

Initialize sum = 0, loop adding a[i] * b[i]

Question 3

What is parallel reduction?

Answer 3

Using binary tree to combine elements: pair, combine, repeat

Question 4

What is the kernel for parallel reduction?

Answer 4

Each work item does partial sum, then binary tree reduction in local memory

Question 5

What is barrier synchronization?

Answer 5

barrier(CLK_LOCAL_MEM_FENCE) - all work items in group wait

Question 6

Why are barriers needed?

Answer 6

Ensure all work items complete writing before reading shared data

Question 7

What is the problem with global barriers?

Answer 7

GPUs cannot synchronize between work groups - no global barrier

Question 8

What is the solution for cross-work-group reduction?

Answer 8

Use multiple kernel launches, each completing before next starts

Question 9

What are lockstep execution?

Answer 9

SIMD cores execute same instruction across all threads simultaneously

Question 10

What is divergence?

Answer 10

When threads in same warp execute different code paths, causing serialization

Question 11

What causes divergence?

Answer 11

if/else branches where different threads take different paths

Question 12

What is the performance impact of divergence?

Answer 12

Code executes serially for different paths, losing parallelism

Question 13

What is subgroup size?

Answer 13

Number of threads executing in lockstep (32 for Nvidia warps, 64 for AMD)

Question 14

Why exploit subgroups in reduction?

Answer 14

Once reduced to subgroup size, no need for explicit synchronization

Question 15

What is the subgroup reduction optimization?

Answer 15

Skip barriers for final reduction steps within subgroup

Question 16

What is the reduction pattern with subgroups?

Answer 16

Normal reduction with barriers until subgroup size, then lockstep reduction

Question 17

What is the advantage of subgroup-aware code?

Answer 17

Reduced synchronization overhead in final reduction steps

Question 18

What is the trade-off with divergence?

Answer 18

Some algorithms accept divergence for cleaner code if benefit outweighs cost

Question 19

What is the key insight about GPU synchronization?

Answer 19

Local synchronization works, global requires multiple kernels

Question 20

What makes GPU synchronization different from CPU?

Answer 20

Work groups are independent, no global coordination primitives