High Foundation Models · 1 min read
Orca: learning GPT-4 reasoning through explanation traces
In one sentence Microsoft Research trains Orca 13B on step-by-step GPT-4 explanations (explanation traces), outperforming ChatGPT on BigBench and AGIEval with 13 billion parameters.
Reading level
Most open-source models learn by imitating only the final answers of larger models. Orca does something different: it also learns the reasoning that leads to the answer, by reading the step-by-step explanations GPT-4 provides when solving a problem.
It's like the difference between copying the math homework and watching how the teacher solves each step on the board.
The result is a 13-billion-parameter model that, on certain complex reasoning tests, beats much larger models including ChatGPT.
Companies
Microsoft
Tools
Orca, GPT-4
Tags
OrcaMicrosoftImitation LearningGPT-4BigBench
Sources