Skip to content
AImpact
IT EN
Inference Intermediate Also known as: Key-Value Cache · Cache chiavi-valori

KV Cache

/kay-vee cache/

A temporary GPU memory that stores attention computations for tokens already seen, so the model does not recompute them on every new token generated.

ShareLinkedInX

In practice

It is why generating the tenth token costs less than the first: the cache avoids redoing work. It eats a lot of VRAM and grows with context, so it is often the real bottleneck for serving many users in parallel. Optimizing it (paged, quantized) is central to cutting inference cost.

Related terms

Attention Transformer Inference compute Paged Attention

Seen in the wild

7 entries mentioning it
  1. DeepSeek V4.1 Flash: 552B open MoE under MIT with KV cache cut to a quarter
    High
  2. FlashInfer 0.2: customizable attention kernels for LLM serving
    Medium
  3. Mooncake: the KV-cache-centric disaggregated architecture serving Kimi
    High
  4. Automatic Prefix Caching in vLLM 0.4.0: KV cache reused across requests with the same prefix
    High
  5. vLLM 0.3.0: FP8 (E5M2) KV cache arrives
    Medium
  6. SGLang: RadixAttention and a structured language, up to 5x throughput
    Medium
  7. vLLM: 24x LLM throughput with PagedAttention from UC Berkeley
    High
← All terms