Also known as: P99, P95, tail latency, latency percentiles
P50 is the median; P95 and P99 are the 95th and 99th percentile latencies. The tail is what wakes oncall, not the median — a 200ms median with a 5s P99 means 1% of users see your system as broken.
In production retrieval, latency is not a single number. It’s a distribution, and the shape of the tail is what determines whether your system feels fast or broken. Two systems with identical 200ms median latency can feel completely different: one with a 250ms P99, one with a 5s P99. The first feels rock-solid; the second has 1% of requests that hang.
A “good” system has the tail close to the median. A 200ms P50 with a 250ms P99 means almost every user has a similar experience. A 200ms P50 with a 5s P99 means a meaningful fraction of users are seeing a different, broken-feeling system.
A retrieval pipeline composed of 4 stages, each with P99 = 500ms (and P50 = 100ms each), has end-to-end P50 of ~400ms — but end-to-end P99 is closer to the sum of P99s, not to 4× the P50. The tail compounds. This is why you can’t reason about compound system latency from medians.
A simple model: if each stage’s slow path fires independently with probability 1%, the chance some stage hits its slow path on a given request is roughly
Latency distributions are typically log-normal or even heavier-tailed: most requests cluster near the median, but a small fraction extend far into the tail. For a heavy-tailed distribution, the mean is dragged up by the rare slow paths and stops reflecting any user’s actual experience — no one waited the mean amount of time. Worse, two operationally different distributions can share an identical mean: a system with uniform 300ms latency and a system with 280ms median plus a 5s P99 both report 320ms mean. Engineers who chase the mean end up optimizing the wrong target. Percentiles are the only summary statistic that survives the heavy-tail shape, which is why every production observability dashboard reports P50, P95, and P99 as first-class.
Optimize the median for cost. Optimize the tail for trust.
In production retrieval pipelines, tail events are typically:
If a single request has 1% chance of slow path, a fan-out request that depends on 10 sub-requests has roughly 10% chance — tail amplification. Retrieval pipelines (BM25 + vector + rerank + LLM) compose 4+ stages; each stage's P99 contributes to the end-to-end P99, not the median. This is why median latency is a near-useless production metric for compound systems.
Depends on the surface. Interactive chat: end-to-end P99 < 3s, with retrieval allocated ~500ms budget. Voice / agent loops: end-to-end P99 < 1s, retrieval < 200ms. Background batch: P99 in seconds is fine. Allocate budget per stage and measure each stage's P99 separately.
GC pauses in the index. Cold-cache reads from disk. Cross-AZ network blips. Embedding model batch backpressure. ANN index merges. Reranker batch starvation when traffic is bursty. Each is silent when measured by averages; each is a P99 tail event. Observability that captures per-stage percentiles is the only way to find them.
