NVIDIA H100 and Hopper architecture: the foundation-model GPU
In one sentence At GTC on March 22, 2022 NVIDIA unveils the Hopper architecture and the H100 GPU: 80 billion transistors, HBM3, Transformer Engine with FP8, 4th-gen NVLink. Availability expected from Q3 2022.
NVIDIA unveils a new generation of data-center GPUs, built to train and run AI models: the H100.
The biggest novelty is the "Transformer Engine", a mix of hardware and software aimed at GPT-style models: it uses 8-bit numbers where they're enough and switches to higher precision where needed, to go faster without losing accuracy.
In the following years the H100 becomes one of the most sought-after GPUs for training large language models, and phrases like "trained on thousands of H100s" become common.
Companies
NVIDIA
Tools
H100, Hopper, DGX H100
Tags
Sources