Skip to content
AImpact
IT EN
Inference Advanced Also known as: Flash Attention

FlashAttention

An algorithm that reorganizes attention computation to minimize data movement between fast and slow GPU memory.

ShareLinkedInX

In practice

It does not change the math, but makes attention much faster and far less memory-hungry. It ships by default in PyTorch and in major inference servers (vLLM, TGI). If you use APIs you never see it; if you self-host it is almost mandatory to turn on.

Related terms

Attention Transformer KV Cache Inference compute

Seen in the wild

4 entries mentioning it
  1. FlashInfer 0.2: customizable attention kernels for LLM serving
    Medium
  2. FlashAttention-3: attention 1.5-2x faster than FA2 on H100, up to 740 TFLOPS in FP16
    High
  3. FlashAttention-2: complete rewrite, about 2x faster, with MQA/GQA and head-dim up to 256
    High
  4. FlashAttention: IO-aware attention that speeds up transformer training
    Landmark
← All terms