NVIDIA A100: Ampere arrives and the GPU that trains GPT-3
In one sentence At GTC 2020 Jensen Huang announces the A100 GPU built on the Ampere architecture: 54 billion transistors, 40-80 GB HBM2e, TF32, 2:4 structured sparsity, and MIG support.
At the heart of every modern AI model is a graphics card. But the "gamer" ones don't cut it: training a model like GPT-3 needs GPUs designed only for data centers.
At GTC 2020, NVIDIA introduces A100, the new generation. It's physically huge (over 50 billion transistors), packs 40 GB of very fast memory, and ships AI-specific tricks: new numeric formats that balance precision and speed, and the ability to "slice" a single GPU into up to seven VMs to share it across workloads.
A100 becomes the AI data-center standard for the next three years. Every major model of 2020-2022 is born here.
Companies
NVIDIA
Tools
A100, Ampere
Tags
Sources