Chinchilla: the big models were undertrained
In one sentence DeepMind publishes the Chinchilla paper and shows that, at equal compute, a smaller model trained on many more tokens wins: Chinchilla (70B) beats Gopher (280B), GPT-3, Jurassic-1 and Megatron-Turing NLG.
Until 2022 the trend was: "let's build the largest possible model", trained on roughly the same amount of text every time. DeepMind argues this is the wrong choice.
The paper shows that with a fixed amount of compute, you're better off with a smaller model fed much more text. To prove it they train Chinchilla, 70 billion parameters, with the same compute used for Gopher (280 billion): Chinchilla beats it almost everywhere, and beats GPT-3 too.
A smaller model is also cheaper to use. The result changes how labs decide how big to make models and how much data to train them on.
Companies
DeepMind
Tools
Chinchilla
Tags
Sources