FLAN: instruction tuning that teaches models to follow directions

Imagine teaching someone to do a generic job: instead of making them repeat the same exercise endlessly, you give them dozens of different tasks and explain each one in plain words. In the end they can adapt to any new request, even without examples.

That's what Google did with FLAN: they took an enormous model (137 billion parameters, LaMDA) and trained it on over 60 types of language tasks, each described as a textual instruction.

The result is striking: the model handles completely new tasks well without ever having seen them during training. Before FLAN, large models were good at completing text but following instructions was a different matter. This experiment is one of the forerunners of the instruction-tuned models that came after it.