Continuous batching
An LLM inference technique that dynamically tops up new requests in an ongoing batch as soon as a slot opens — instead of waiting until the whole batch is done. It maximizes GPU utilization and throughput in an LLM serving engine.
Problem with regular (static) batching: different requests generate different numbers of tokens, so they finish at different times; a static batch must wait for the longest → the GPU sits partly idle. Continuous batching (also "in-flight" or "dynamic" batching) treats each generation step as an opportunity to pull in new waiting requests into the gaps that open as others finish → the GPU is kept saturated. Win: dramatically higher throughput (several times) in serving systems. Popularized by vLLM (together with PagedAttention) and now in TGI, TensorRT-LLM, and others. A core technique behind efficient LLM serving at scale. Related to PagedAttention and KV cache.