A persisting L2 access policy window over a 64 MB weight matrix

A bandwidth-bound GEMV kernel is launched thousands of times on one stream, and every launch reads the same 64 MB weight matrix from start to end. The GPU is a 40 GB A100 (40 MB of L2, not in MIG mode), and the 64 MB window is within its accessPolicyMaxWindowSize. The engineer reserves 30 MB of L2 (three quarters of it) for persisting accesses and sets the stream's access policy window as shown, expecting roughly half the matrix to be served from L2 on each launch. Nsight Compute shows the L2 hit rate on the matrix is near zero, no better than with no window at all. What is going on?

setup.cu
cudaDeviceSetLimit(cudaLimitPersistingL2CacheSize, 30 << 20);

cudaStreamAttrValue attr = {};
attr.accessPolicyWindow.base_ptr  = d_weights;
attr.accessPolicyWindow.num_bytes = 64 << 20;     // whole matrix
attr.accessPolicyWindow.hitRatio  = 1.0f;
attr.accessPolicyWindow.hitProp   = cudaAccessPropertyPersisting;
attr.accessPolicyWindow.missProp  = cudaAccessPropertyStreaming;
cudaStreamSetAttribute(stream, cudaStreamAttributeAccessPolicyWindow, &attr);

38 of 46 in GPU / CUDA