FP8
8-bit floating-point format for deep learning — half the size of FP16, double the tensor throughput on Nvidia Hopper/Ada and later.
Two variants (NVIDIA/Arm/Intel spec): E4M3 (4 exponent, 3 mantissa — better precision, for weights and activations) and E5M2 (5 exponent, 2 mantissa — better range, for gradients). Per-tensor scaling keeps dynamics manageable.
Supported in H100, H200, B200 (FP8 + FP4) and MI300X. In practice: 2–3× faster training than BF16 with comparable quality if scaling is done right. Common in new frontier runs (GPT-4o, Llama 3.1+) and in vLLM/TensorRT-LLM serving.