Medium
Allocating and freeing a scratch buffer inside a multi-stream loop
This loop pipelines chunks across four non-default streams using pinned host buffers, so copies and kernels from different chunks should overlap. Each iteration needs a temporary device buffer between stage1 and stage2. How does the per-iteration cudaMalloc/cudaFree affect the pipeline, and what is the idiomatic fix?
pipeline.cu
for (int i = 0; i < numChunks; ++i) {
cudaStream_t s = streams[i % 4];
size_t off = i * chunkElems;
float* scratch;
cudaMalloc(&scratch, chunkBytes);
cudaMemcpyAsync(d_in + off, h_in + off, chunkBytes, cudaMemcpyHostToDevice, s);
stage1<<<grid, block, 0, s>>>(d_in + off, scratch);
stage2<<<grid, block, 0, s>>>(scratch, d_out + off);
cudaMemcpyAsync(h_out + off, d_out + off, chunkBytes, cudaMemcpyDeviceToHost, s);
cudaFree(scratch);
}34 of 46 in GPU / CUDA