And you are here why?

Let’s take for granted that you’ve done the right thing — you’ve generously instrumented your system, and are actually paying attention to the metrics that you’ve generated (•). The question on the table is — “ Do you actually trust the metrics that you are generating? ” (Hint: You shouldn’t) Let’s look at something fairly straightforward, the request/response path as shown below You would think that the ResponseTime would be the sum of each of the processing stages, right? i.e.,  ResponseTime = 10 + 1 + 20 + 1 + 5 = 37ms ? But, since you shouldn’t trust your metrics, you 1) Also measured  ResponseTime  directly, and 2) Compared it against the what it should be, and 3) Charted/alerted on deviations and you found that the actual ResponseTime was, say,  52ms . That’s quite a difference, no? As to  Why  it was 52ms, let’s look at a bunch of possible issues 1.  Measuring the Wrong Thing : You actually instrumented something completely diffe...

“ You can use logging for everything! ”  —  #CowboyDeveloper Well, technically speaking, the resident  #CowboyDeveloper  is correct, you  can  use logging for everything • Metrics : Print out your data, and cut/paste them from the terminal into Google Sheets. Voila! You can now generate all the charts you need from this data! ( I’m not joking. I know a dude that actually did this! ) • Alerting :  tail -f | grep  on your logs, and if anything  bad  happens, pipe it to sendmail. ( Yes. sendmail was a thing. Don’t ask ) • Monitoring :  tail -f | grep  on your logs across many, many terms. ( One for each server, times the number of things you are looking for ) • Debugging : You know the drill. Throw enough  print()  statements in there, and, with enough hair-pulling, you can figure out the problem. • Tracing : Uhhh, what’s that? ( And, once explained, “solved” by sticking unique IDs into each log statement. Again, don’...

You think I'd have learned by now, but the following rules are ones that always hit me upside the head, even though I know better! I'm not saying that these rules are inviolable (this is software after all!). It's just that if you violate the rule, be sure to have an excellent reason involved, and even then, check again, and then document it. The documentation, in particular, will help you remind yourself what a butthead you are… 1. Make Scaling intrinsic to the design . It doesn't matter whether this means autoscale groups, k8s, or auto-instantiation of domain objects, the point here is that scale - at all levels - can not, and should not, be something that you bolt on afterwards. 2. Make Deployment a first class citizen . Ideally, a merge into master should trigger a production deployment. And yes, this implies code coverage, comprehensive testing, toolchains, and everything else you've (hopefully!) read about. Maybe you're not there yet, but you rea...

The good news around metrics is that it is increasingly easy to instrument and monitor your code, with plenty of usable software out there like  Prometheus ,  TICK ,  OpenTracing , and whatnot. The bad news is, well, pretty much the same — it’s so easy to instrument and monitor your code that the sheer amount of stuff that you can capture is mind-boggling. It is  so  easy to get caught up in all the charts and graphs that you will find yourself constantly losing sight of the the forest for the trees. All that data ticking around on the screen can, counter-intuitively, mean that it is actually  harder  to actually get any information (let alone knowledge!) from the metrics. Latency Latency is a particularly good example of this. Most everyone out there measures this as an  average , and while that’s better than nothing, it can easily hide issues. The fault, as always, lies in us — in that we tend to think about our systems in terms of steady-...

I already covered  “ Does It Work? ” from the customer’s perspective , looking at  R ate ( Does it scale? ),  E rrors ( Are these the responses the customer needs? ), and  D uration ( How quick was the response? ) which together go by the acronym  RED . The thing is  these metrics are great for the customers currently using the service, but they give you limited information about potential customers!  Think of it this way — if your system is right at it’s limit (whatever that means), then your current customers might be very happy, but the next one that hops on might get terrible service. Or worse, the service or  everybody  might crap out!. From an  observability  perspective, you should be looking — at the very least — at • Utilization  (aka “ How busy is it? ”): The percentage of time the service is busy doing stuff, and conversely, the percentage of time it has available to do more stuff. • Saturation  (aka: “...

Words have meanings, and using these terms interchangeably — which we all tend to do — can lead to serious long term issues. Lets start with definitions. Technically speaking,   “use   measure   for more concrete or objective attributes and   metric for more abstract, higher-level, or somewhat subjective attributes” (•). To make this more concrete, in software terms, measures are the things that you should be instrumenting in your code, things like   queue length ,   request count , and so forth, while metrics are the higher-level abstractions like   failure rate ,   number of bugs , or even   customer acquisition cost . The problem with conflating the terms is that eventually,   we end up implementing the wrong thing . When done well, your code/system should have a separation of concerns, viz.  • Code instrumented to track the measures. Think counters, gauges, etc. (   #Prometheus ! ) to track queue length/duration, AP...

You’ve really gotten into this whole Observability thing, and have started plugging Prometheus into, well, everything that doesn’t already have it.  And you’ve   also   started implementing OpenTracing because, well, you’ve got a distributed something, and not doing it, or something like it, would be just plain dumb. And you realize that you’ve now spent the last week pulling together some kind of consistent wire protocol that works seamlessly across the metrics and the traces,   and you haven’t even started on the APIs , and surely,   surely somebody has already done this?   Enter OpenCensus (•), the open-sourced version of Google’s Census library, and which gives you  • Standard wire protocols and consistent APIs for handling trace and metric data.  • A single set of libraries for many languages, including Erlang, Java, C++, Go, Python, Ruby, and more  • Integrations with your favorite web and RPC frameworks  • Exporters for storage...