vLLM: 24x LLM throughput with PagedAttention from UC Berkeley
In one sentence A UC Berkeley team releases vLLM, an open source LLM serving library built on PagedAttention, which manages the KV cache like OS virtual memory: up to 24x the throughput of HuggingFace Transformers and up to 3.5x that of TGI.
When using large language models, there's a hidden problem: the memory used for intermediate calculations (the KV cache) is managed very inefficiently. vLLM solves this by borrowing the same idea operating systems use to manage RAM.
Instead of reserving contiguous memory blocks (which often remain partially empty), vLLM splits the KV cache into small "pages" and allocates them wherever space is available — exactly like Linux virtual memory. This nearly eliminates all waste.
The result: the same GPU can serve many more requests in parallel, with throughput up to 24 times higher than HuggingFace Transformers. Before the public release, vLLM was already the engine serving Vicuna and LMSYS's Chatbot Arena.
Companies
UC Berkeley
Tools
vLLM, PagedAttention
Tags
Sources