Skip to content
AImpact
IT EN
Training Beginner Also known as: Pre-training · Pre-addestramento

Pretraining

The initial training phase where a model learns the structure of language by predicting the next token on huge amounts of generic text.

ShareLinkedInX

In practice

It is the most expensive step (months of GPUs and millions of dollars) and produces a "base" model that can write but cannot yet follow instructions. Only big labs run it from scratch; companies start from pretrained models and adapt them with SFT, LoRA, or RLHF.

Related terms

Foundation model SFT Loss Function Gradient Descent Checkpoint

Seen in the wild

6 entries mentioning it
  1. GR-2: ByteDance pre-trains a robot on 38,000 hours of human internet videos
    High
  2. UL2: Google unifies pretraining paradigms with Mixture-of-Denoisers
    Medium
  3. The Pile: the 825 GB open dataset that fuels the open LLM era
    High
  4. The Pile: the open-source 825 GB dataset for training LLMs
    High
  5. Image GPT: generative pretraining for images
    Medium
  6. ELECTRA: more efficient NLP pre-training than BERT
    Medium
← All terms