Efficient Multi-Scale Deformable Attention on GPUs
On this page
Multi-scale deformable attention (MSDA) is the operator that makes DETR-style detectors affordable to run, and the part of them that runs slowest. This paper measures where the time actually goes on two generations of NVIDIA GPU, and uses that to guide new CUDA and Triton kernels for the operator.
- Dispatch-order reordering yields no speedupacross seven query orders, three sampling-point counts and four level counts, no order is statistically distinguishable from linear — L2 locality is set by the query-block tiling, not by the order queries are dispatched in
- High occupancy is not high throughputan 85%-occupancy tiling reaches 5.1% of A100 peak bandwidth while a 17%-occupancy tiling reaches 36.1% and runs 7.4 times faster
- Emulated BF16 atomics bottleneck the backward passAmpere has no native half-precision atomic add, so the gradient scatter becomes a compare-and-swap retry loop; an FP32 accumulator removes the bottleneck entirely
The kernels are available in Deformops, a Python library — see installation and usage below.
Summary
DETR recast detection as set prediction with a transformer, removing anchors and non-maximum suppression. Deformable DETR made it affordable.
Context & background
MSDA is the core operator of DETR-family detectors and their efficiency bottleneck: in Deformable DETR the MSDA encoder accounts for 49% of GFLOPs but only 11% of average precision. Dense attention is fast because its queries, keys and values sit in contiguous memory; MSDA samples at offsets predicted by a learned projection, so the addresses it reads are unknown until the kernel runs — scattered loads on the way in, scattered atomic accumulation on the way out.
Occupancy is the fraction of a GPU's warp slots kept resident. Morton and Hilbert orders are space-filling curves, which keep nearby cells nearby in memory.
Two heuristics that do not apply
Reordering queries so that neighbours sample neighbouring memory changes nothing: across seven dispatch orders, no order is statistically distinguishable from plain linear dispatch, because the locality that matters is created by the tiling inside a query block. Occupancy misleads in the same way — the tiling that keeps five times as many warps resident moves a seventh as many bytes per second, since the limit is the size distribution of DRAM transactions and not the availability of warps.
The backward pass is an atomics problem
The gradient scatter is where training time goes, and its cost is decided by whether the hardware has an atomic add for the accumulator's data type. Before Hopper it does not, so a BF16 atomic add compiles to a roughly ten-instruction compare-and-swap loop that retries on every concurrent write — the bottleneck at encoder scale, and invisible to any FLOP-based metric. Accumulating in FP32 routes through a native instruction and closes the gap on A100, which makes the backward choice a decision about accumulator precision rather than about CUDA versus Triton.
The full measurements — roofline metrics, latency and memory at both scales, the ablations and the end-to-end training numbers — are in the paper.
Library
Deformops provides MSDA as PyTorch operators and modules. It includes a Triton and a CUDA implementation, both built on the findings of this paper, and installs from PyPI:
pip install deformopsCitation
If you find this work useful, please consider citing the paper.
@article{stolle2026efficient,
title = {Efficient Multi-Scale Deformable Attention on GPUs},
author = {Stolle, Kurt H. W.},
journal = {Transactions on Machine Learning Research},
year = {2026},
url = {https://openreview.net/forum?id=Q4jZ7zKNKx}
}This publication is part of the NEON project with file number 17628 of the Crossover research program, which is (partly) financed by the Dutch Research Council (NWO). The Dutch national compute infrastructure was used with the support of the SURF Cooperative using grant EINF-5438.