Afterburner
Michael Sargis · 2024 · Research project
A GPU kernel compiler born out of a personal grudge against training progress bars.
The idea
The model was waiting on memory. The memory was waiting on a layout conversion. I was waiting on both.
Afterburner searches for fused kernels that remove intermediate reads and writes. Each candidate has to pass numerical checks before it gets a benchmark run. Fast and wrong is still wrong.
Results
On our lab workload, the compiler delivered an 11× end-to-end training speedup. That number belongs to this workload. It is not a claim about every transformer or every GPU.
What survived the experiment
Measure the full run, including compilation, data movement, and the parts your beautiful kernel does not touch. Keep the slow reference implementation. You will need it at 2 a.m.