Groq LPU: 500-tokens-per-second inference goes viral
In one sentence In mid-February 2024 Groq's public demo on open models such as Mixtral 8x7B generates about 500 tokens per second and goes viral on X; GroqCloud launches for developers in the same days.
With almost every chatbot you wait for text to appear word by word. Groq, a startup founded in 2016 by former Google staff led by Jonathan Ross (one of the TPU designers), built a chip dedicated to language-model inference that it calls the LPU (Language Processing Unit).
In February 2024 its public demo, running open models such as Mixtral 8x7B, generates answers at about 500 tokens per second: the reply appears almost all at once. Tests spread on X and Groq becomes a talking point within days.
Inference speed, until then taken for granted as slow, becomes something to compete on.
Companies
Groq
Tools
Groq LPU, GroqCloud
Tags
Sources