Mixture of Denoisers
A pretraining strategy (UL2, Google 2022) that trains a single model on multiple denoising objectives simultaneously: left-to-right language modeling, span prediction (BERT-style masked spans of varying lengths and corruptions), and prefix language modeling. Unifies the strengths of GPT-style and T5-style pretraining. The model learns when to use each mode based on a sentinel token that signals the objective type.
In practice
A researcher wanting a flexible model for both completion and question answering can use UL2 or a Flan-UL2 checkpoint without choosing between encoder-decoder (T5) and decoder-only (GPT) architectures. In practice, the sentinel token `[S2S]`, `[NLU]`, or `[NLG]` must be prepended to the prompt to activate the correct mode — a detail that significantly impacts performance and is often omitted, causing poor results.
Related terms
Seen in the wild
115 entries mentioning it- HighZ.ai opens GLM-5.3-Flash: mystery model Ox Alpha revealed, MIT-licensed 320B MoE
- HighOpenAI launches the GPT-5.6 family in three variants: Luna, Terra, and Sol
- LandmarkMeta releases Llama 4.1: Scout, Maverick, and Behemoth MoE models under Apache 2.0
- HighGoogle I/O 2026: Gemini Ultra 3, Project Astra goes live on Pixel, 2M context with real-time grounding, Veo 3.2, Imagen 4
- LandmarkAnthropic releases Claude Fable 5: a new model family beyond the 4.x line
- MediumRealtime voice AI: sub-second latency and multilingual become the norm
- HighBoston Dynamics Atlas Electric: manipulation foundation model trained on 10 million robot hours
- MediumMistral releases Devstral Small: 7B coding model for agentic tasks on consumer GPU
- HighMCP at 18 months: the server ecosystem hits critical mass
- HighRobotics foundation model: a new step toward the "GPT of manipulation"
- HighMistral Small 4: three models (reasoning + vision + coding) fused into one open weight
- MediumGroq launches GroqCloud 2.0: LPU Gen3, 2000 tokens/sec, and Frankfurt European data center
- MediumNano Banana 2: Google rebuilds its viral image model around consistency and text
- HighAlibaba releases Qwen 3.5: sparse multimodal MoE, 262K context, Apache 2.0
- HighDeepSeek releases Janus Pro: one model to understand and generate images
- HighGemini 3 Pro and Flash: Google relaunches the frontier challenge
- LandmarkAlibaba releases Qwen2.5-VL 72B: best open-source multimodal model beats GPT-4o on key benchmarks
- HighOpenAI releases o3-mini: advanced reasoning at a fraction of the cost
- HighMCP ecosystem 2025: Inspector, UI, registry, and cross-vendor adoption
- MediumClaude Haiku 4.5: the small model that matches May's Sonnet 4