“Does It Work?” — The View From Inside

I already covered “Does It Work?” from the customer’s perspective, looking at Rate (Does it scale?), Errors (Are these the responses the customer needs?), and Duration (How quick was the response?) which together go by the acronym RED.

The thing is these metrics are great for the customers currently using the service, but they give you limited information about potential customers! Think of it this way — if your system is right at it’s limit (whatever that means), then your current customers might be very happy, but the next one that hops on might get terrible service. Or worse, the service or everybody might crap out!.

From an observability perspective, you should be looking — at the very least — at

• • Utilization (aka “How busy is it?”): The percentage of time the service is busy doing stuff, and conversely, the percentage of time it has available to do more stuff.
• • Saturation (aka: “What is the backlog?”): How much extra work does the service have that it can’t deal with — this is often (usually?) queued.
• • Errors (aka: “How frequently is it barfing?”: Literally keeping track of the number of times that the service barfs.

Incidentally, do note that these aren’t just about averages — distributions and percentiles really matter here. For example, in a bursty environment the average utilization might be low, but the performance might be quite sucky if each time there is a burst of traffic, the system saturates, and it takes a while for things to come back to normal.

Anyhow, put together, these commonly go by the acronym USE (Utilization, Saturation, Errors). As Baron Schwartz puts it “taken together, RED and USE comprise minimally complete, maximally useful observability — a way to understand both aspects of a system: its users/customers and the work they request, as well as its resources/components and how they react to the workload.”

In short, if you want meaningfully maintain and control the behavior of your your systems, you need to implement USE + RED — or some variation thereof like LTES (•). If you don’t, well, you’re flying blind!

(•) LTES (Latency, Traffic, Errors, Saturation) isthe SRE version of the above — ref. “The Four Golden Signals” .