Automatic Prefix Caching in vLLM 0.4.0: KV cache reused across requests with the same prefix

Before generating the first word, a model must process all the context text: system prompt, history, retrieved documents. This phase is called prefill.

With Automatic Prefix Caching, if a new request starts with the same text as a request already processed, vLLM reuses the work already done on that part instead of redoing it. It helps when many requests share a long system prompt or the same document.

The gain only concerns context processing time, not the speed at which the answer's words are generated.