A100
Nvidia's data center GPU from 2020, built on the Ampere architecture. The workhorse that trained the first generation of genuinely large language models — GPT-3 was trained on tens of thousands of them.
54 billion transistors on 7 nm, 40 or 80 GB of HBM2e memory with up to 2 TB/s of bandwidth, third-generation tensor cores. Two innovations mattered: the TF32 format, which gave a large speedup without any code changes, and MIG (Multi-Instance GPU), which partitions one card into up to seven isolated instances — ideal for inference and shared clusters. NVLink connects cards within a server at far higher bandwidth than PCIe. Succeeded by the H100 (Hopper, 2022) and H200, but A100s remain all over the clouds because they're still perfectly usable and now cheaper per hour.