# ZYFA: Speed and Accuracy of FP8 FlashAttention on AMD CDNA4

DevFeed: [ZYFA: Speed and Accuracy of FP8 FlashAttention on AMD CDNA4](<https://devfeed.tech/articles/zyfa-speed-and-accuracy-of-fp8-flashattention-on-amd-cdna4-120624.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07496>)

Author: Quentin Anthony, Ganesh Bikshandi, Praneeth Medepalli, Stephen Ebert

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [rocm](<https://devfeed.tech/topics/rocm.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>)

Tags: [amd](<https://devfeed.tech/tags/amd.md>), [amd-instinct-mi355x](<https://devfeed.tech/tags/amd-instinct-mi355x.md>), [attention](<https://devfeed.tech/tags/attention.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [error](<https://devfeed.tech/tags/error.md>), [flashattention](<https://devfeed.tech/tags/flashattention.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [hopper](<https://devfeed.tech/tags/hopper.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [matrix](<https://devfeed.tech/tags/matrix.md>), [ping-pong](<https://devfeed.tech/tags/ping-pong.md>), [power](<https://devfeed.tech/tags/power.md>), [prefill](<https://devfeed.tech/tags/prefill.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [schedule](<https://devfeed.tech/tags/schedule.md>), [speed](<https://devfeed.tech/tags/speed.md>)

## AI overview

ZYFA's MI355X FP8 attention prefill kernels outperform AITER in the reported tests while reducing numerical error at head dimension 128. They replace exponentials in probability conversion with a multiply-add and integer conversion, overlap softmax and matrix work with a ping-pong thread schedule, and scale probability blocks by powers of two before rounding and normalization. On inputs captured from three models, ZYFA reaches 1.16-1.76x AITER throughput; Gaussian kernel benchmarks at 32K tokens show 1.26-1.37x at head dimension 128 and 1.99x for DeepSeek-style multi-head latent attention. At head dimension 128, its numerical error is within a few percent of Composable Kernel and Triton and 62-78% lower than AITER's. Results on random inputs may not predict real-activation throughput because MI355X sustained clock speed varies with operand values.

## Source excerpt

arXiv:2611.07496v1 Announce Type: cross Abstract: Attention is a core building block of frontier models and often dominates compute-bound prefill at long context. FP8 matrix instructions offer twice the peak throughput of BF16, but they accelerate only matrix multiplication, and the softmax between the two matrix multiplications of attention runs on vector units that FP8 does not accelerate. On the AMD Instinct MI355X, the vendor's AITER kernels reach 39% of peak FP8 matrix throughput at 32K tokens, compared with 49% in BF16. Attention is also more numerically sensitive to quantization than feed-forward layers, and unlike NVIDIA's Hopper and Blackwell GPUs, the MI355X does not track the completion of matrix instructions in hardware, so overlapping the softmax with the matrix multiplications depends on instruction order. We present ZYFA, prefill kernels for the MI355X built on three techniques: (1) computing each FP8 probability directly from its score with a multiply-add and an integer conversion instead of an exponential; (2) a ping-pong schedule in which one group of threads computes its softmax while another executes its matrix multiplications; and (3) scaling each block of probabilities up by a power of two before rounding, and normalizing by the sum of the rounded probabilities. On attention inputs captured from three models and timed at sustained clock speeds, ZYFA achieves 1.16-1.76x the throughput of AITER. On the Gaussian inputs of kernel benchmarks at 32K tokens, it achieves 1.26-1.37x at head dimension 128 and 1.99x for DeepSeek-style multi-head latent attention. The sustained clock of the MI355X depends on operand values, so throughput on random inputs does not reliably predict throughput on real model activations. At head dimension 128, ZYFA's numerical error is within a few percent of that of Composable Kernel and Triton and 62-78% lower than that of AITER's. The source code is available at https://github.com/Zyphra/zyfa.