What Is Latency?

Latency is not response time

Latency is the wait before data starts flowing; response time is the whole round trip including the transfer. For a small JSON payload the two are nearly identical, which is why they get used interchangeably — and for a 5 MB download they are nothing alike.

Keeping them separate makes results diagnosable. High latency with fast transfer points at distance, queueing, or slow server processing. Low latency with slow transfer points at payload size, compression, or bandwidth. Collapsing both into one number throws that signal away.

Where latency comes from

Three contributions, and under load they behave very differently from each other.

• Propagation delay — the speed of light over distance. Roughly 30 ms each way between London and New York, and no amount of optimisation changes it.
• Processing delay — the server actually doing the work before it can answer.
• Queueing delay — waiting for a worker, a thread, or a connection pool slot. This is the one that explodes under load.

Why load raises latency

Propagation is fixed and processing is roughly constant per request, so almost all the latency growth in a load test is queueing. That is why the curve is not linear: as utilisation approaches 100%, queueing delay rises sharply, and a system at 95% utilisation has dramatically worse latency than the same system at 70% doing only slightly less work.

It also explains why measuring from the wrong place misleads. A load generator in the same data centre as the target reports a network path no real customer has, and it will show latency numbers your users will never see. Generate load from the regions your users are actually in.