Skip to content
AImpact
IT EN
Inference Advanced Also known as: Decoding speculativo

Speculative Decoding

A technique where a small fast model proposes several tokens ahead and the large model verifies them in a single pass, accepting the correct ones.

ShareLinkedInX

In practice

It can produce answers 2-3x faster with no change in final quality, because the big model stays the judge. It is used in production by OpenAI, Anthropic, and in self-hosted runtimes. It needs a "draft" model aligned with the main one, so it is not free to set up.

Related terms

Inference compute Distillation Greedy Decoding Logits

Seen in the wild

3 entries mentioning it
  1. llama.cpp: speculative decoding with draft models for 2-3x speedup
    High
  2. Medusa: multi-head speculative decoding without a separate draft model, 2.2x speedup
    High
  3. Speculative Decoding: 2-3x LLM inference speedup without changing output
    High
← All terms