Also known as: tokens per second, tps, tok/s
Tokens per second per GPU is the production planning metric for LLM serving. Throughput scales with batch size up to a memory-bound ceiling, then plateaus. The key number for capacity planning, autoscaling, and unit-economic analysis.
Throughput is the production planning metric for LLM serving. Specifically: tokens generated per second per GPU. Everything from how many GPUs you need, to how much each token costs, to how long users wait — derives from this number.
Order of magnitude, on H100s in 2026:
Different stacks (vLLM, TGI, TensorRT-LLM, SGLang) hit different points; benchmarks vary by ~30% between them on identical hardware.
LLM inference is memory-bound. For each forward pass, the GPU has to load tens of GB of weights from HBM into compute units; the actual matrix multiplies are fast in comparison. With batch 1, you load all weights to produce one token. With batch 32, you load the same weights and produce 32 tokens — same memory traffic, 32× the work.
This is why throughput scales nearly linearly with batch size up to a ceiling, then plateaus. The ceiling is set by:
The honest method is empirical: spin up the inference engine, ramp concurrent requests, and watch the throughput curve. The shape is consistent — a roughly linear region, a knee, then a plateau (sometimes followed by a small dip as scheduler overhead grows).
The analytic shortcut: model bandwidth-bound throughput as
The cleanest way to push the ceiling is to lower
A request has two phases:
Production throughput numbers usually mean decode throughput, since prefill is rarely the bottleneck. But long prompts with short outputs are prefill-dominated, and you need different tuning. Vendors increasingly report prefill and decode separately.
Bigger batches improve throughput but worsen per-request latency — each request waits in queue longer, and competes for the same KV cache memory. The fundamental tradeoff:
Continuous batching (vLLM-style) finds a much better Pareto frontier than naive static batching by letting requests join and leave the in-flight batch dynamically.
A reranker is called on every retrieval, so its tokens/sec/GPU sets the number of concurrent users per GPU dollar. Sub-1B specialized cross-encoders sit in the sweet spot: big enough to encode strong relevance signal, small enough that an H100 sustains thousands of relevance judgments per second.
Different requests have different token counts (prompt + generation). A query with a 100-token answer is much cheaper than one with 1000. Tokens/sec normalizes against this; requests/sec doesn't. For capacity planning and unit economics, tokens/sec is what aligns with cost (you pay for GPU time, which produces tokens).
Memory bandwidth, almost always. LLM inference is memory-bound — most GPU time is spent moving weights from HBM to compute units, not computing. Batching amortizes this across requests, but eventually you saturate the bandwidth or run out of VRAM for the KV cache. That's the ceiling.
Partially. Bigger batches give more throughput but worse per-request latency (each request waits longer in the batch). Continuous batching and speculative decoding are the techniques that improve one without ruining the other. Production serving stacks (vLLM, TGI, TensorRT-LLM) put substantial engineering into the Pareto frontier.
