← the index · class 07 · metrics

The average hides the tail

The dashboard says the service responds in 80 ms on average. Users complain that it is slow. Who is right?

Both. This is the catalogue's GET /products/{id}, over the last 1 000 requests, loaded 1 000 at a time. Every number below is computed from the requests that were sent.

79.4 ms
average — what the dashboard shows
20 ms
median
26 ms
95th percentile
3.85 s
99th percentile — the users who complain
15
over 1 s (1.5 %)
Each press adds 1 000 product reads; the numbers stay put, because the mix does not change.

Why: the distribution

The same requests in the buckets of the histogram that /metrics exports. Almost all of them are in the first buckets; a handful are out in the seconds. The average falls between the two, where there is no request at all. The 95th percentile does not see the tail either: it is 1.5 % of the requests, less than 5 %. The 99th does.

≤ 5 ms
0
≤ 10 ms
0
≤ 25 ms
904
≤ 50 ms
81
≤ 75 ms
0
≤ 100 ms
0
≤ 250 ms
0
≤ 500 ms
0
≤ 750 ms
0
≤ 1.00 s
0
≤ 2.50 s
0
≤ 5.00 s
15
≤ 7.50 s
0
≤ 10.00 s
0
> 10 s
0

Split by route

«All routes» is the single number of a dashboard nobody split: for the catalogue it is 79.3 ms on average, and 3.85 s at the 99th percentile. Every status has its own row — successes apart from failures.

ServiceRouteStatusRequestsAveragep50p95p99Max
gateway all routes 1469 ms469 ms469 ms469 ms469 ms
gateway POST /orders201 1469 ms469 ms469 ms469 ms469 ms
orders all routes 1457 ms457 ms457 ms457 ms457 ms
orders POST /orders201 1457 ms457 ms457 ms457 ms457 ms
catalogue all routes 1 00179.3 ms20 ms26 ms3.85 s4.15 s
catalogue GET /products/{id}200 1 00079.4 ms20 ms26 ms3.85 s4.15 s
catalogue POST /reservations200 19 ms9 ms9 ms9 ms9 ms
payments all routes 1410 ms410 ms410 ms410 ms410 ms
payments POST /charges201 1410 ms410 ms410 ms410 ms410 ms

Requests: since start-up, as in /metrics. Average and percentiles: over the last durations of each series (up to 10 000), nearest rank. The route is the pattern, never the path: one series for every product.

Metrics to alert, logs to explain

The metric cannot say which product is slow — product_id is not an attribute, or there would be one series per product. The log can: the catalogue writes a WARN line for every read over one second.

product_idSlow reads in the log
3115

The lines themselves →