llama.cpp: speculative decoding with a draft model comes to llama-server
In one sentence On November 25, 2024 Georgi Gerganov merges PR #10455: llama-server, the llama.cpp HTTP server, supports speculative decoding with a draft model. In the PR's tests, Qwen2.5-Coder 32B with a 0.5B draft goes from 29 to 47 tokens/s on an RTX 3090 (about 1.6x).
llama.cpp is the open-source library that lets AI models run on ordinary hardware. Speculative decoding already existed as an example program; with this change it comes to llama-server, the server that exposes models over an API and the one used in practice.
The principle: a small, fast model (the draft) proposes a few tokens ahead, and the large model checks them all at once in a single pass. When the proposals are right time is saved, and the final answer stays the one the large model would have given.
In the tests published in the PR the gain ranges from about 1.25x to 1.7x depending on GPU and models.
Companies
ggml.ai
Tools
llama.cpp, llama-server
Tags
Sources