And you are here why?

Let’s take for granted that you’ve done the right thing — you’ve generously instrumented your system, and are actually paying attention to the metrics that you’ve generated (•). The question on the table is — “ Do you actually trust the metrics that you are generating? ” (Hint: You shouldn’t) Let’s look at something fairly straightforward, the request/response path as shown below You would think that the ResponseTime would be the sum of each of the processing stages, right? i.e.,  ResponseTime = 10 + 1 + 20 + 1 + 5 = 37ms ? But, since you shouldn’t trust your metrics, you 1) Also measured  ResponseTime  directly, and 2) Compared it against the what it should be, and 3) Charted/alerted on deviations and you found that the actual ResponseTime was, say,  52ms . That’s quite a difference, no? As to  Why  it was 52ms, let’s look at a bunch of possible issues 1.  Measuring the Wrong Thing : You actually instrumented something completely diffe...

The good news around metrics is that it is increasingly easy to instrument and monitor your code, with plenty of usable software out there like  Prometheus ,  TICK ,  OpenTracing , and whatnot. The bad news is, well, pretty much the same — it’s so easy to instrument and monitor your code that the sheer amount of stuff that you can capture is mind-boggling. It is  so  easy to get caught up in all the charts and graphs that you will find yourself constantly losing sight of the the forest for the trees. All that data ticking around on the screen can, counter-intuitively, mean that it is actually  harder  to actually get any information (let alone knowledge!) from the metrics. Latency Latency is a particularly good example of this. Most everyone out there measures this as an  average , and while that’s better than nothing, it can easily hide issues. The fault, as always, lies in us — in that we tend to think about our systems in terms of steady-...

I already covered  “ Does It Work? ” from the customer’s perspective , looking at  R ate ( Does it scale? ),  E rrors ( Are these the responses the customer needs? ), and  D uration ( How quick was the response? ) which together go by the acronym  RED . The thing is  these metrics are great for the customers currently using the service, but they give you limited information about potential customers!  Think of it this way — if your system is right at it’s limit (whatever that means), then your current customers might be very happy, but the next one that hops on might get terrible service. Or worse, the service or  everybody  might crap out!. From an  observability  perspective, you should be looking — at the very least — at • Utilization  (aka “ How busy is it? ”): The percentage of time the service is busy doing stuff, and conversely, the percentage of time it has available to do more stuff. • Saturation  (aka: “...

You’re ingesting measures (not metrics.  Measures aren’t Metrics — know the difference !), and, clearly, you’re time-stamping stuff, right? Well, let’s think about that for a moment. What exactly  does  “time-stamping” mean? More to the point,  what exactly are you time-stamping ? Consider the following Creation Time : Theoretically, the time at which that data was created. Question is,  whose wall-clock are we looking at ? Unless this is a single monolith (and even there, you might be screwing up time measurements 😱), you’re looking at a distributed system, and, well, everything is relative at that point. (•) Ingestion Time : The time  you  received the data. Again, this is true for a given value of “you”, after all, “you” might be a distributed system too. Storage Time : The time when you actually stored the data that you ingested. And the reason you might care about this is because you might make, or have made, decisions based on data,...

Every looked at your metrics, and wondered “ WhyTF am I actually tracking these things ”? Well yeah, there are a whole bunch of obvious “Technology” metrics — you know, the things you use to track failures, latencies, and so forth. This question has more to do with your “Process” metrics — where you are measuring things like burn-down, open tickets, time to resolution, etc. The thing is, some of these are easy to measure, and some of them require quite a bit of work just so you can get to a place where you can start tracking them. However,   all   of these are pointless if you can’t directly tie them to decisions! Think of it this way — if the decisions are orthogonal to the metrics (“ we’re going to fire Alice regardless of what the burn-down rate is ”), then why even bother tracking the metric? It’s a really simple question to ask, and one that can be surprisingly difficult to answer — What are the thing that you would   do more of ,   do less of ,   a...

Words have meanings, and using these terms interchangeably — which we all tend to do — can lead to serious long term issues. Lets start with definitions. Technically speaking,   “use   measure   for more concrete or objective attributes and   metric for more abstract, higher-level, or somewhat subjective attributes” (•). To make this more concrete, in software terms, measures are the things that you should be instrumenting in your code, things like   queue length ,   request count , and so forth, while metrics are the higher-level abstractions like   failure rate ,   number of bugs , or even   customer acquisition cost . The problem with conflating the terms is that eventually,   we end up implementing the wrong thing . When done well, your code/system should have a separation of concerns, viz.  • Code instrumented to track the measures. Think counters, gauges, etc. (   #Prometheus ! ) to track queue length/duration, AP...