Hard
An atomic-counter barrier between two phases of one kernel
To avoid a second kernel launch, an engineer separates two phases with a hand-rolled grid-wide barrier. On an A100 (108 SMs), the occupancy calculator says this kernel fits 8 blocks per SM. Launched with 64 blocks it works every time. Launched with 4096 blocks, the kernel never finishes. Why, and what is the supported way to get this structure?
kernel.cu
__device__ unsigned int arrived = 0;
__global__ void twoPhase(float* a, int n) {
phase1(a, n);
__syncthreads();
if (threadIdx.x == 0) {
__threadfence();
atomicAdd(&arrived, 1);
while (atomicAdd(&arrived, 0) < gridDim.x) { } // spin until all blocks arrive
}
__syncthreads();
phase2(a, n);
}37 of 46 in GPU / CUDA