Emu3: a single transformer for text, images and video using only next-token prediction
In one sentence On September 27, 2024 BAAI's Emu3 team publishes Emu3: images, text and video are converted into discrete tokens and a single 8B transformer, trained from scratch with next-token prediction only, handles both generation and understanding.
Image generation usually relies on diffusion models, and image understanding usually connects a vision encoder to a language model. Emu3 tries a simpler route: it turns images and video into discrete "tokens", like words in a visual vocabulary, and trains a single transformer to predict the next token, exactly as is done with text.
According to the authors, this approach outperforms established task-specific models such as SDXL for image generation and LLaVA-1.6 for understanding, and it can also generate video by predicting tokens one after another.
Companies
BAAI
Tools
Emu3
Tags
Sources