Flan-T5 and Flan-PaLM: instruction tuning scales to 1,800 tasks
In one sentence Google scales instruction tuning to 1,836 tasks and up to 540B parameters: Flan-PaLM 540B beats PaLM 540B by 9.4% on average and reaches 75.2% on five-shot MMLU. The Flan-T5 checkpoints are released publicly.
After 2021's FLAN — which used 60 task types — Google massively raised the bar: about 1,800 different tasks, drawn from several dataset collections. It's like going from a basic course to a full university curriculum.
An important part is chain-of-thought: for some tasks, instead of just providing correct answers, the training data includes step-by-step reasoning. The model learns not just what to answer, but how to think through it.
Google then released Flan-T5, a family of models ranging from 80 million to 11 billion parameters, with public weights. The smaller sizes are light enough to run on ordinary hardware.
Companies
Tools
—
Tags
Sources