SGLang: RadixAttention and a structured language, up to 5x throughput
In one sentence LMSYS introduces SGLang, a language and runtime for structured LLM programs: RadixAttention automatically reuses the KV cache across different calls, with up to 5x higher throughput than Guidance and vLLM in the blog's tests.
Many AI applications send the model requests that start the same way: system instructions, examples, conversation history. Normally the engine recomputes even the parts it has already seen.
On January 17, 2024 the LMSYS team introduces SGLang, which keeps computations already done (the KV cache) and reuses them when a new request shares its beginning with an earlier one. In the tests published on the blog, on Llama-7B and Mixtral-8x7B, SGLang reaches up to 5x higher throughput than systems such as Guidance and vLLM.
Besides the engine, SGLang offers a small language embedded in Python for writing programs that call the model several times, in parallel or with constraints on the output.
Companies
LMSYS
Tools
SGLang, RadixAttention
Tags
Sources