DeepSpeed-FastGen: Dynamic SplitFuse for LLM serving with up to 2.3x vLLM's throughput
In one sentence On November 3, 2023 Microsoft's DeepSpeed team introduces FastGen, an LLM serving system built on DeepSpeed-MII and DeepSpeed-Inference. With Dynamic SplitFuse scheduling it claims up to 2.3x effective throughput versus vLLM and up to 3.7x lower tail latency.
When an AI service receives many requests at once, the server must handle prompts of very different lengths. In continuous-batching systems like vLLM, processing a long prompt can briefly pause generation for responses already in flight, and users see slowdowns.
DeepSpeed-FastGen introduces "Dynamic SplitFuse": it splits long prompts into pieces spread across several model passes and merges short ones, so each pass carries a similar amount of work and generation does not stall.
According to the DeepSpeed team, the result is up to 2.3x vLLM's effective throughput, roughly half the average latency and up to 3.7x lower per-token tail latency. It installs with the deepspeed-mii package and is released as an alpha.
Companies
Microsoft
Tools
DeepSpeed, DeepSpeed-MII, DeepSpeed-FastGen, vLLM
Tags
Sources