BLT Byte Latent Transformer
Meta's December 2024 architecture that drops the tokenizer entirely and instead groups raw bytes into dynamically sized patches according to how predictable the text is.
A fixed vocabulary carries well-known problems: under-represented languages get absurd token counts, typos and rare characters shatter into fragments, and the model cannot see the letters inside a word. Running on raw bytes instead is too expensive, since sequences get roughly four times longer. BLT resolves this with a small model that estimates the entropy of the next byte and puts a patch boundary wherever uncertainty spikes — predictable text is packed into long patches, difficult material into short ones. Compute therefore lands where it is needed. Scaling runs showed parity with Llama 3 at matched training budget, with better robustness to noisy input.